2022-09-27T15:36:23.0700441Z Requested labels: linux.8xlarge.nvidia.gpu
2022-09-27T15:36:23.0700526Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/85462/merge
2022-09-27T15:36:23.0700548Z Waiting for a runner to pick up this job...
2022-09-27T15:36:41.6163092Z Job is about to start running on the runner: i-037610acd6d6b81a9 (organization)
2022-09-27T15:36:46.1222191Z Current runner version: '2.296.2'
2022-09-27T15:36:46.1230206Z Runner name: 'i-037610acd6d6b81a9'
2022-09-27T15:36:46.1230981Z Runner group name: 'Default'
2022-09-27T15:36:46.1231713Z Machine name: 'ip-10-0-4-203'
2022-09-27T15:36:46.1234545Z ##[group]GITHUB_TOKEN Permissions
2022-09-27T15:36:46.1235501Z Actions: read
2022-09-27T15:36:46.1235938Z Checks: read
2022-09-27T15:36:46.1236315Z Contents: read
2022-09-27T15:36:46.1236746Z Deployments: read
2022-09-27T15:36:46.1237229Z Discussions: read
2022-09-27T15:36:46.1237605Z Issues: read
2022-09-27T15:36:46.1238038Z Metadata: read
2022-09-27T15:36:46.1238456Z Packages: read
2022-09-27T15:36:46.1238818Z Pages: read
2022-09-27T15:36:46.1239289Z PullRequests: read
2022-09-27T15:36:46.1239782Z RepositoryProjects: read
2022-09-27T15:36:46.1240546Z SecurityEvents: read
2022-09-27T15:36:46.1241024Z Statuses: read
2022-09-27T15:36:46.1241457Z ##[endgroup]
2022-09-27T15:36:46.1245841Z Secret source: None
2022-09-27T15:36:46.1246730Z Prepare workflow directory
2022-09-27T15:36:46.2579376Z Prepare all required actions
2022-09-27T15:36:46.2808299Z Getting action download info
2022-09-27T15:36:46.4999078Z Download action repository 'pytorch/pytorch@master' (SHA:15c52ffc4f9a02f7078033677d44ccd760107952)
2022-09-27T15:36:49.8810020Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-09-27T15:36:50.0111839Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:3c1d75049465d7dfa70acca6d80b9c5c06ff4886)
2022-09-27T15:36:50.3114777Z Getting action download info
2022-09-27T15:36:50.5161682Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246)
2022-09-27T15:36:50.7320916Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-09-27T15:36:50.7323453Z ##[group] Inputs
2022-09-27T15:36:50.7323838Z   build-environment: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:36:50.7325013Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
]}

2022-09-27T15:36:50.7326267Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:36:50.7326745Z   sync-tag: 
2022-09-27T15:36:50.7326973Z ##[endgroup]
2022-09-27T15:36:50.7327788Z Complete job name: linux-bionic-cuda11.6-py3.10-gcc7 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu)
2022-09-27T15:36:50.8428976Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-09-27T15:36:50.8429391Z with:
2022-09-27T15:36:50.8429626Z   submodules: recursive
2022-09-27T15:36:50.8429898Z   fetch-depth: 0
2022-09-27T15:36:50.8430136Z env:
2022-09-27T15:36:50.8430368Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:36:50.8430635Z ##[endgroup]
2022-09-27T15:36:50.8734231Z ##[group]Run retry () {
2022-09-27T15:36:50.8734546Z retry () {
2022-09-27T15:36:50.8734859Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-09-27T15:36:50.8735156Z }
2022-09-27T15:36:50.8735398Z echo "${GITHUB_WORKSPACE}"
2022-09-27T15:36:50.8735699Z if [ -z "${NO_SUDO}" ]; then
2022-09-27T15:36:50.8736012Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-09-27T15:36:50.8736294Z else
2022-09-27T15:36:50.8736552Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-09-27T15:36:50.8737006Z fi
2022-09-27T15:36:50.8737316Z mkdir "${GITHUB_WORKSPACE}"
2022-09-27T15:36:50.8754918Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:36:50.8755251Z env:
2022-09-27T15:36:50.8755505Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:36:50.8755748Z   NO_SUDO: 
2022-09-27T15:36:50.8755987Z ##[endgroup]
2022-09-27T15:36:50.8986362Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:36:51.0416190Z ##[group]Run malfet/checkout@silent-checkout
2022-09-27T15:36:51.0416484Z with:
2022-09-27T15:36:51.0416754Z   ref: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:36:51.0417035Z   fetch-depth: 0
2022-09-27T15:36:51.0417271Z   submodules: recursive
2022-09-27T15:36:51.0417534Z   quiet-checkout: true
2022-09-27T15:36:51.0417871Z   repository: pytorch/pytorch
2022-09-27T15:36:51.0418324Z   token: ***
2022-09-27T15:36:51.0418563Z   ssh-strict: true
2022-09-27T15:36:51.0418834Z   persist-credentials: true
2022-09-27T15:36:51.0419111Z   clean: true
2022-09-27T15:36:51.0419335Z   lfs: false
2022-09-27T15:36:51.0419590Z   set-safe-directory: true
2022-09-27T15:36:51.0419840Z env:
2022-09-27T15:36:51.0420062Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:36:51.0420320Z ##[endgroup]
2022-09-27T15:36:51.1961470Z Syncing repository: pytorch/pytorch
2022-09-27T15:36:51.1963554Z ##[group]Getting Git version info
2022-09-27T15:36:51.1964123Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-09-27T15:36:51.1964745Z [command]/usr/bin/git version
2022-09-27T15:36:51.1965009Z git version 2.37.1
2022-09-27T15:36:51.1972997Z ##[endgroup]
2022-09-27T15:36:51.1995035Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/5aa188a6-3cd6-4142-ad20-2e6c2e629769' before making global git config changes
2022-09-27T15:36:51.1995627Z Adding repository directory to the temporary git global config as a safe directory
2022-09-27T15:36:51.2003933Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:36:51.2047512Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-09-27T15:36:51.2053159Z ##[group]Initializing the repository
2022-09-27T15:36:51.2059442Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:36:51.2092291Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-09-27T15:36:51.2093262Z hint: is subject to change. To configure the initial branch name to use in all
2022-09-27T15:36:51.2094020Z hint: of your new repositories, which will suppress this warning, call:
2022-09-27T15:36:51.2094343Z hint: 
2022-09-27T15:36:51.2094715Z hint: 	git config --global init.defaultBranch <name>
2022-09-27T15:36:51.2095194Z hint: 
2022-09-27T15:36:51.2095736Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-09-27T15:36:51.2096820Z hint: 'development'. The just-created branch can be renamed via this command:
2022-09-27T15:36:51.2097383Z hint: 
2022-09-27T15:36:51.2097828Z hint: 	git branch -m <name>
2022-09-27T15:36:51.2098359Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-09-27T15:36:51.2108057Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-09-27T15:36:51.2143641Z ##[endgroup]
2022-09-27T15:36:51.2144645Z ##[group]Disabling automatic garbage collection
2022-09-27T15:36:51.2149741Z [command]/usr/bin/git config --local gc.auto 0
2022-09-27T15:36:51.2181814Z ##[endgroup]
2022-09-27T15:36:51.2182748Z ##[group]Setting up auth
2022-09-27T15:36:51.2192631Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-09-27T15:36:51.2230863Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-09-27T15:36:51.2552737Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-09-27T15:36:51.2586854Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-09-27T15:36:51.2888116Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-09-27T15:36:51.3048822Z ##[endgroup]
2022-09-27T15:36:51.3049750Z ##[group]Fetching the repository
2022-09-27T15:36:51.3058319Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-09-27T15:37:39.6752929Z [command]/usr/bin/git rev-parse --verify --quiet 52424e2bf38e454d535881fed9628d3e20f4f944^{object}
2022-09-27T15:37:39.6793390Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:37:42.8100988Z ##[endgroup]
2022-09-27T15:37:42.8101625Z ##[group]Determining the checkout info
2022-09-27T15:37:42.8102860Z ##[endgroup]
2022-09-27T15:37:42.8103878Z ##[group]Checking out the ref
2022-09-27T15:37:42.8109997Z [command]/usr/bin/git checkout --quiet --force 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:37:44.5010482Z ##[endgroup]
2022-09-27T15:37:44.5011119Z ##[group]Setting up auth for fetching submodules
2022-09-27T15:37:44.5018550Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-09-27T15:37:44.5072383Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-09-27T15:37:44.5106867Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-09-27T15:37:44.5140672Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-09-27T15:37:44.5170454Z ##[endgroup]
2022-09-27T15:37:44.5171355Z ##[group]Fetching submodules
2022-09-27T15:37:44.5176100Z [command]/usr/bin/git submodule sync --recursive
2022-09-27T15:37:44.5500066Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-09-27T15:37:44.5805115Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-09-27T15:37:44.5807111Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-09-27T15:37:44.5810275Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-09-27T15:37:44.5813435Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-09-27T15:37:44.5816761Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-09-27T15:37:44.5820636Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-09-27T15:37:44.5823957Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-09-27T15:37:44.5827688Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-09-27T15:37:44.5831574Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-09-27T15:37:44.5835644Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-09-27T15:37:44.5839847Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-09-27T15:37:44.5845075Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-09-27T15:37:44.5849570Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-09-27T15:37:44.5853954Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-09-27T15:37:44.5858791Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-09-27T15:37:44.5863771Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-09-27T15:37:44.5868392Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-09-27T15:37:44.5873516Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:37:44.5878543Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-09-27T15:37:44.5884575Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-09-27T15:37:44.5889920Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-09-27T15:37:44.5895646Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-09-27T15:37:44.5901244Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-09-27T15:37:44.5907137Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-09-27T15:37:44.5913009Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-09-27T15:37:44.5919147Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-09-27T15:37:44.5925962Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-09-27T15:37:44.5932164Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-09-27T15:37:44.5938741Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-09-27T15:37:44.5945272Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-09-27T15:37:44.5952077Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-09-27T15:37:44.5958769Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-09-27T15:37:44.5966627Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-09-27T15:37:44.5973747Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-09-27T15:37:44.5980897Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-09-27T15:37:44.5988244Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-09-27T15:37:44.5995697Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-09-27T15:37:44.6004306Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-09-27T15:37:44.6011961Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-09-27T15:37:44.6020194Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-09-27T15:37:44.6028169Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-09-27T15:37:44.6056301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-09-27T15:37:44.9505812Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-09-27T15:37:45.2116470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-09-27T15:37:45.4296271Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-09-27T15:37:45.7570250Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-09-27T15:37:46.2176222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-09-27T15:37:48.3920655Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-09-27T15:37:53.5154369Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-09-27T15:37:54.1742035Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-09-27T15:37:54.7527914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-09-27T15:37:56.2777883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-09-27T15:37:57.6546250Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-09-27T15:37:59.0937669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-09-27T15:38:08.5919835Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-09-27T15:38:09.3069296Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-09-27T15:38:10.6127811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-09-27T15:38:11.8002485Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-09-27T15:38:12.0756861Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-09-27T15:38:12.6107968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-09-27T15:38:12.9471010Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-09-27T15:38:13.9241180Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-09-27T15:38:14.3546756Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-09-27T15:38:14.5491541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-09-27T15:38:15.3556433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-09-27T15:38:17.0938485Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-09-27T15:38:17.8001811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-09-27T15:38:18.1955975Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-09-27T15:38:25.0707842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-09-27T15:38:26.7348692Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-09-27T15:38:27.1775038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-09-27T15:38:27.4821652Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-09-27T15:38:33.3667566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-09-27T15:38:33.5509572Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-09-27T15:38:33.7742180Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-09-27T15:38:34.6483912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-09-27T15:38:34.8777123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-09-27T15:38:35.2360780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-09-27T15:38:35.5618111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-09-27T15:38:36.1223823Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-09-27T15:38:38.8087658Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-09-27T15:38:39.3055294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-09-27T15:38:41.6378942Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-09-27T15:38:41.6502320Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-09-27T15:38:41.6595536Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-09-27T15:38:41.6878269Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-09-27T15:38:41.7150992Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-09-27T15:38:41.7619863Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-09-27T15:38:42.5377223Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-09-27T15:38:42.5622332Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-09-27T15:38:42.6845414Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-09-27T15:38:42.7246200Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-09-27T15:38:43.0798163Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-09-27T15:38:43.5999441Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-09-27T15:38:43.9169394Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-09-27T15:38:43.9732776Z Submodule path 'third_party/fbgemm': checked out '499cd22f5c2e26041e4f190f628b48478a89a030'
2022-09-27T15:38:43.9749216Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:38:43.9752349Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:38:43.9755571Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:38:43.9759175Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:38:43.9786047Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-09-27T15:38:45.1008669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-09-27T15:38:45.7009172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-09-27T15:38:46.8923259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-09-27T15:38:47.3071044Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-09-27T15:38:47.4292980Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-09-27T15:38:47.4979572Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-09-27T15:38:47.5089224Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-09-27T15:38:47.6134316Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-09-27T15:38:47.6524251Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-09-27T15:38:47.6621603Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-09-27T15:38:47.7088246Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-09-27T15:38:47.7366959Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72'
2022-09-27T15:38:47.7913192Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-09-27T15:38:47.8039715Z Submodule path 'third_party/ideep': checked out '77d662b313a762e82b389d3fd965e0098f12cd99'
2022-09-27T15:38:47.8055582Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-09-27T15:38:47.8081417Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-09-27T15:38:55.1904679Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-09-27T15:38:55.1924079Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:38:55.1951107Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-09-27T15:39:02.7465153Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-09-27T15:39:02.7581486Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-09-27T15:39:02.7747934Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-09-27T15:39:02.8874545Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573'
2022-09-27T15:39:02.8892348Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:02.8895542Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:02.8921985Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-09-27T15:39:04.1252397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-09-27T15:39:05.2182216Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-09-27T15:39:05.2822692Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-09-27T15:39:05.3057747Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-09-27T15:39:05.3209708Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-09-27T15:39:05.4502370Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-09-27T15:39:05.7556652Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-09-27T15:39:05.7589210Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:05.7592675Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:05.7619844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-09-27T15:39:06.1807220Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-09-27T15:39:07.0293642Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-09-27T15:39:07.0672882Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-09-27T15:39:07.0848770Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-09-27T15:39:07.0865321Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:07.0891770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-09-27T15:39:08.7705530Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-09-27T15:39:08.7728165Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:08.7731265Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:08.7757962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-09-27T15:39:09.2202587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-09-27T15:39:10.0546689Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-09-27T15:39:10.1303791Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-09-27T15:39:10.1320566Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:10.1347574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-09-27T15:39:10.3990492Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-09-27T15:39:10.4092865Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-09-27T15:39:10.7197479Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-09-27T15:39:10.7219203Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:10.7222063Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:10.7249910Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-09-27T15:39:11.1315151Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-09-27T15:39:12.3642002Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-09-27T15:39:12.4461135Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-09-27T15:39:12.4556100Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-09-27T15:39:12.4683881Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-09-27T15:39:12.5078719Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020'
2022-09-27T15:39:12.5177609Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-09-27T15:39:12.5511944Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-09-27T15:39:12.5616447Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-09-27T15:39:12.6149115Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-09-27T15:39:12.7479926Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-09-27T15:39:12.7785492Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-09-27T15:39:12.7803800Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:12.7806707Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:12.7809827Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:12.7813159Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:12.7840251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-09-27T15:39:13.8331359Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-09-27T15:39:14.1466182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-09-27T15:39:15.4246855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-09-27T15:39:16.9975715Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-09-27T15:39:17.0151050Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-09-27T15:39:17.0954919Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-09-27T15:39:17.1285089Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-09-27T15:39:17.1301868Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:17.1328916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-09-27T15:39:18.2157081Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-09-27T15:39:18.3742092Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-09-27T15:39:18.3776957Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-09-27T15:39:18.4099905Z Entering 'android/libs/fbjni'
2022-09-27T15:39:18.4142215Z Entering 'third_party/FP16'
2022-09-27T15:39:18.4185536Z Entering 'third_party/FXdiv'
2022-09-27T15:39:18.4226780Z Entering 'third_party/NNPACK'
2022-09-27T15:39:18.4269258Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:18.4311496Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:18.4354300Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:18.4407675Z Entering 'third_party/benchmark'
2022-09-27T15:39:18.4448971Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:18.4491374Z Entering 'third_party/cub'
2022-09-27T15:39:18.4533876Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:18.4581343Z Entering 'third_party/cutlass'
2022-09-27T15:39:18.4629696Z Entering 'third_party/eigen'
2022-09-27T15:39:18.4674609Z Entering 'third_party/fbgemm'
2022-09-27T15:39:18.4717155Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:18.4757953Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:18.4799219Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:18.4842747Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:18.4884464Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:18.4929890Z Entering 'third_party/fmt'
2022-09-27T15:39:18.4972908Z Entering 'third_party/foxi'
2022-09-27T15:39:18.5014624Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:18.5056036Z Entering 'third_party/gloo'
2022-09-27T15:39:18.5097674Z Entering 'third_party/googletest'
2022-09-27T15:39:18.5139336Z Entering 'third_party/ideep'
2022-09-27T15:39:18.5181711Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:18.5225349Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:18.5273500Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:18.5315004Z Entering 'third_party/ittapi'
2022-09-27T15:39:18.5356776Z Entering 'third_party/kineto'
2022-09-27T15:39:18.5398480Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:18.5441839Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:18.5484965Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:18.5527697Z Entering 'third_party/neon2sse'
2022-09-27T15:39:18.5568058Z Entering 'third_party/nlohmann'
2022-09-27T15:39:18.5610305Z Entering 'third_party/onnx'
2022-09-27T15:39:18.5664874Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:18.5705994Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:18.5749020Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:18.5790051Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:18.5835843Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:18.5877682Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:18.5919107Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:18.5965323Z Entering 'third_party/pocketfft'
2022-09-27T15:39:18.6006597Z Entering 'third_party/protobuf'
2022-09-27T15:39:18.6051659Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:18.6092510Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:18.6136098Z Entering 'third_party/psimd'
2022-09-27T15:39:18.6176809Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:18.6217548Z Entering 'third_party/pybind11'
2022-09-27T15:39:18.6259613Z Entering 'third_party/python-enum'
2022-09-27T15:39:18.6300639Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:18.6341647Z Entering 'third_party/python-six'
2022-09-27T15:39:18.6383016Z Entering 'third_party/sleef'
2022-09-27T15:39:18.6424217Z Entering 'third_party/tbb'
2022-09-27T15:39:18.6467491Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:18.6508811Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:18.6551605Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:18.6591993Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:18.6634137Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:18.6676367Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:18.6722274Z Entering 'third_party/zstd'
2022-09-27T15:39:18.6772574Z ##[endgroup]
2022-09-27T15:39:18.6773410Z ##[group]Persisting credentials for submodules
2022-09-27T15:39:18.6781823Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-09-27T15:39:18.7088466Z Entering 'android/libs/fbjni'
2022-09-27T15:39:18.7128971Z Entering 'third_party/FP16'
2022-09-27T15:39:18.7169420Z Entering 'third_party/FXdiv'
2022-09-27T15:39:18.7211322Z Entering 'third_party/NNPACK'
2022-09-27T15:39:18.7252137Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:18.7293329Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:18.7334933Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:18.7385168Z Entering 'third_party/benchmark'
2022-09-27T15:39:18.7425991Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:18.7468133Z Entering 'third_party/cub'
2022-09-27T15:39:18.7509333Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:18.7556762Z Entering 'third_party/cutlass'
2022-09-27T15:39:18.7605155Z Entering 'third_party/eigen'
2022-09-27T15:39:18.7648996Z Entering 'third_party/fbgemm'
2022-09-27T15:39:18.7690779Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:18.7730356Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:18.7771117Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:18.7811052Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:18.7853520Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:18.7896533Z Entering 'third_party/fmt'
2022-09-27T15:39:18.7937465Z Entering 'third_party/foxi'
2022-09-27T15:39:18.7979569Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:18.8020517Z Entering 'third_party/gloo'
2022-09-27T15:39:18.8061891Z Entering 'third_party/googletest'
2022-09-27T15:39:18.8102868Z Entering 'third_party/ideep'
2022-09-27T15:39:18.8142995Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:18.8184906Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:18.8231434Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:18.8272675Z Entering 'third_party/ittapi'
2022-09-27T15:39:18.8315048Z Entering 'third_party/kineto'
2022-09-27T15:39:18.8355933Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:18.8396470Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:18.8438461Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:18.8480016Z Entering 'third_party/neon2sse'
2022-09-27T15:39:18.8520050Z Entering 'third_party/nlohmann'
2022-09-27T15:39:18.8562496Z Entering 'third_party/onnx'
2022-09-27T15:39:18.8618008Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:18.8658997Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:18.8702565Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:18.8743581Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:18.8789367Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:18.8831428Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:18.8871502Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:18.8917729Z Entering 'third_party/pocketfft'
2022-09-27T15:39:18.8958565Z Entering 'third_party/protobuf'
2022-09-27T15:39:18.9004008Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:18.9045701Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:18.9088227Z Entering 'third_party/psimd'
2022-09-27T15:39:18.9128783Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:18.9169701Z Entering 'third_party/pybind11'
2022-09-27T15:39:18.9211085Z Entering 'third_party/python-enum'
2022-09-27T15:39:18.9251961Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:18.9292360Z Entering 'third_party/python-six'
2022-09-27T15:39:18.9332923Z Entering 'third_party/sleef'
2022-09-27T15:39:18.9373894Z Entering 'third_party/tbb'
2022-09-27T15:39:18.9416460Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:18.9457624Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:18.9498178Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:18.9538286Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:18.9579977Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:18.9619575Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:18.9662924Z Entering 'third_party/zstd'
2022-09-27T15:39:18.9719237Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-09-27T15:39:19.0029857Z Entering 'android/libs/fbjni'
2022-09-27T15:39:19.0069980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-09-27T15:39:19.0087404Z Entering 'third_party/FP16'
2022-09-27T15:39:19.0125715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-09-27T15:39:19.0142560Z Entering 'third_party/FXdiv'
2022-09-27T15:39:19.0180938Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-09-27T15:39:19.0198130Z Entering 'third_party/NNPACK'
2022-09-27T15:39:19.0236766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-09-27T15:39:19.0254235Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:19.0291778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-09-27T15:39:19.0308779Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:19.0347475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-09-27T15:39:19.0365462Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:19.0403524Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-09-27T15:39:19.0430870Z Entering 'third_party/benchmark'
2022-09-27T15:39:19.0469701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:39:19.0487430Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:19.0525534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-09-27T15:39:19.0542965Z Entering 'third_party/cub'
2022-09-27T15:39:19.0581870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-09-27T15:39:19.0598967Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:19.0637910Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-09-27T15:39:19.0661264Z Entering 'third_party/cutlass'
2022-09-27T15:39:19.0698713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-09-27T15:39:19.0722268Z Entering 'third_party/eigen'
2022-09-27T15:39:19.0759403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-09-27T15:39:19.0778696Z Entering 'third_party/fbgemm'
2022-09-27T15:39:19.0817706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-09-27T15:39:19.0835056Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:19.0873434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-09-27T15:39:19.0890785Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:19.0929401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-09-27T15:39:19.0946553Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:19.0985341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:19.1002839Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:19.1039506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-09-27T15:39:19.1057650Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:19.1095962Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-09-27T15:39:19.1114885Z Entering 'third_party/fmt'
2022-09-27T15:39:19.1152764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-09-27T15:39:19.1170829Z Entering 'third_party/foxi'
2022-09-27T15:39:19.1209722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-09-27T15:39:19.1226838Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:19.1265183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-09-27T15:39:19.1283990Z Entering 'third_party/gloo'
2022-09-27T15:39:19.1321011Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-09-27T15:39:19.1338416Z Entering 'third_party/googletest'
2022-09-27T15:39:19.1375725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:19.1392574Z Entering 'third_party/ideep'
2022-09-27T15:39:19.1431575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-09-27T15:39:19.1448436Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:19.1486642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-09-27T15:39:19.1505014Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:19.1543279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-09-27T15:39:19.1567301Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:19.1605202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-09-27T15:39:19.1621805Z Entering 'third_party/ittapi'
2022-09-27T15:39:19.1659768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-09-27T15:39:19.1676524Z Entering 'third_party/kineto'
2022-09-27T15:39:19.1714410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-09-27T15:39:19.1731823Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:19.1769428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-09-27T15:39:19.1787062Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:19.1824444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:19.1843696Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:19.1880956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-09-27T15:39:19.1898225Z Entering 'third_party/neon2sse'
2022-09-27T15:39:19.1936707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-09-27T15:39:19.1953346Z Entering 'third_party/nlohmann'
2022-09-27T15:39:19.1992137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-09-27T15:39:19.2011493Z Entering 'third_party/onnx'
2022-09-27T15:39:19.2049588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-09-27T15:39:19.2079238Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:19.2117750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:39:19.2135931Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:19.2174291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:39:19.2193313Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:19.2231858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-09-27T15:39:19.2249411Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:19.2288006Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-09-27T15:39:19.2309791Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:19.2349078Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:39:19.2366755Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:19.2405632Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:39:19.2423455Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:19.2462344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-09-27T15:39:19.2484525Z Entering 'third_party/pocketfft'
2022-09-27T15:39:19.2521982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-09-27T15:39:19.2540173Z Entering 'third_party/protobuf'
2022-09-27T15:39:19.2578682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-09-27T15:39:19.2598961Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:19.2636998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-09-27T15:39:19.2654348Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:19.2691196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:19.2709697Z Entering 'third_party/psimd'
2022-09-27T15:39:19.2748232Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-09-27T15:39:19.2765583Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:19.2803968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-09-27T15:39:19.2820612Z Entering 'third_party/pybind11'
2022-09-27T15:39:19.2858692Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:39:19.2875706Z Entering 'third_party/python-enum'
2022-09-27T15:39:19.2913004Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-09-27T15:39:19.2929847Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:19.2968268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-09-27T15:39:19.2985212Z Entering 'third_party/python-six'
2022-09-27T15:39:19.3023256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-09-27T15:39:19.3040598Z Entering 'third_party/sleef'
2022-09-27T15:39:19.3078565Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-09-27T15:39:19.3095272Z Entering 'third_party/tbb'
2022-09-27T15:39:19.3133572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-09-27T15:39:19.3152143Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:19.3190274Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-09-27T15:39:19.3207733Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:19.3245140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-09-27T15:39:19.3262004Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:19.3300712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-09-27T15:39:19.3317162Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:19.3356254Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-09-27T15:39:19.3373942Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:19.3411104Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-09-27T15:39:19.3427367Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:19.3466271Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-09-27T15:39:19.3486401Z Entering 'third_party/zstd'
2022-09-27T15:39:19.3524685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-09-27T15:39:19.4448080Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-09-27T15:39:19.4754578Z Entering 'android/libs/fbjni'
2022-09-27T15:39:19.4796517Z Entering 'third_party/FP16'
2022-09-27T15:39:19.4840303Z Entering 'third_party/FXdiv'
2022-09-27T15:39:19.4881870Z Entering 'third_party/NNPACK'
2022-09-27T15:39:19.4923873Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:19.4966266Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:19.5007524Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:19.5060130Z Entering 'third_party/benchmark'
2022-09-27T15:39:19.5101999Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:19.5145906Z Entering 'third_party/cub'
2022-09-27T15:39:19.5187322Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:19.5235704Z Entering 'third_party/cutlass'
2022-09-27T15:39:19.5283837Z Entering 'third_party/eigen'
2022-09-27T15:39:19.5328414Z Entering 'third_party/fbgemm'
2022-09-27T15:39:19.5370176Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:19.5411088Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:19.5453055Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:19.5494755Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:19.5536664Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:19.5581671Z Entering 'third_party/fmt'
2022-09-27T15:39:19.5622598Z Entering 'third_party/foxi'
2022-09-27T15:39:19.5664038Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:19.5706109Z Entering 'third_party/gloo'
2022-09-27T15:39:19.5749130Z Entering 'third_party/googletest'
2022-09-27T15:39:19.5790657Z Entering 'third_party/ideep'
2022-09-27T15:39:19.5831687Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:19.5874918Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:19.5923536Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:19.5964822Z Entering 'third_party/ittapi'
2022-09-27T15:39:19.6007073Z Entering 'third_party/kineto'
2022-09-27T15:39:19.6048699Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:19.6089924Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:19.6132539Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:19.6174409Z Entering 'third_party/neon2sse'
2022-09-27T15:39:19.6215800Z Entering 'third_party/nlohmann'
2022-09-27T15:39:19.6257720Z Entering 'third_party/onnx'
2022-09-27T15:39:19.6312528Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:19.6354256Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:19.6398090Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:19.6439778Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:19.6487157Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:19.6529691Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:19.6571378Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:19.6618603Z Entering 'third_party/pocketfft'
2022-09-27T15:39:19.6659557Z Entering 'third_party/protobuf'
2022-09-27T15:39:19.6704920Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:19.6747255Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:19.6790220Z Entering 'third_party/psimd'
2022-09-27T15:39:19.6832174Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:19.6873943Z Entering 'third_party/pybind11'
2022-09-27T15:39:19.6915864Z Entering 'third_party/python-enum'
2022-09-27T15:39:19.6957706Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:19.6999586Z Entering 'third_party/python-six'
2022-09-27T15:39:19.7041455Z Entering 'third_party/sleef'
2022-09-27T15:39:19.7082996Z Entering 'third_party/tbb'
2022-09-27T15:39:19.7126114Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:19.7168457Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:19.7210666Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:19.7252511Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:19.7294987Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:19.7335992Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:19.7381410Z Entering 'third_party/zstd'
2022-09-27T15:39:19.7437002Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-09-27T15:39:19.7743603Z Entering 'android/libs/fbjni'
2022-09-27T15:39:19.7785938Z Entering 'third_party/FP16'
2022-09-27T15:39:19.7827846Z Entering 'third_party/FXdiv'
2022-09-27T15:39:19.7868798Z Entering 'third_party/NNPACK'
2022-09-27T15:39:19.7910849Z Entering 'third_party/QNNPACK'
2022-09-27T15:39:19.7953372Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T15:39:19.7995261Z Entering 'third_party/XNNPACK'
2022-09-27T15:39:19.8048308Z Entering 'third_party/benchmark'
2022-09-27T15:39:19.8089563Z Entering 'third_party/cpuinfo'
2022-09-27T15:39:19.8132290Z Entering 'third_party/cub'
2022-09-27T15:39:19.8173773Z Entering 'third_party/cudnn_frontend'
2022-09-27T15:39:19.8220490Z Entering 'third_party/cutlass'
2022-09-27T15:39:19.8268755Z Entering 'third_party/eigen'
2022-09-27T15:39:19.8313619Z Entering 'third_party/fbgemm'
2022-09-27T15:39:19.8356459Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T15:39:19.8397335Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T15:39:19.8439804Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T15:39:19.8482130Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T15:39:19.8524192Z Entering 'third_party/flatbuffers'
2022-09-27T15:39:19.8568093Z Entering 'third_party/fmt'
2022-09-27T15:39:19.8610518Z Entering 'third_party/foxi'
2022-09-27T15:39:19.8652717Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T15:39:19.8693926Z Entering 'third_party/gloo'
2022-09-27T15:39:19.8736353Z Entering 'third_party/googletest'
2022-09-27T15:39:19.8779331Z Entering 'third_party/ideep'
2022-09-27T15:39:19.8820409Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T15:39:19.8864819Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T15:39:19.8913198Z Entering 'third_party/ios-cmake'
2022-09-27T15:39:19.8955128Z Entering 'third_party/ittapi'
2022-09-27T15:39:19.8996449Z Entering 'third_party/kineto'
2022-09-27T15:39:19.9038607Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T15:39:19.9081250Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T15:39:19.9124416Z Entering 'third_party/nccl/nccl'
2022-09-27T15:39:19.9166870Z Entering 'third_party/neon2sse'
2022-09-27T15:39:19.9208436Z Entering 'third_party/nlohmann'
2022-09-27T15:39:19.9252697Z Entering 'third_party/onnx'
2022-09-27T15:39:19.9307304Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T15:39:19.9348913Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T15:39:19.9393539Z Entering 'third_party/onnx-tensorrt'
2022-09-27T15:39:19.9435414Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T15:39:19.9482551Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T15:39:19.9525368Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T15:39:19.9567617Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T15:39:19.9613221Z Entering 'third_party/pocketfft'
2022-09-27T15:39:19.9653841Z Entering 'third_party/protobuf'
2022-09-27T15:39:19.9698313Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T15:39:19.9739483Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T15:39:19.9782968Z Entering 'third_party/psimd'
2022-09-27T15:39:19.9824974Z Entering 'third_party/pthreadpool'
2022-09-27T15:39:19.9866789Z Entering 'third_party/pybind11'
2022-09-27T15:39:19.9909071Z Entering 'third_party/python-enum'
2022-09-27T15:39:19.9950047Z Entering 'third_party/python-peachpy'
2022-09-27T15:39:19.9991628Z Entering 'third_party/python-six'
2022-09-27T15:39:20.0033142Z Entering 'third_party/sleef'
2022-09-27T15:39:20.0075235Z Entering 'third_party/tbb'
2022-09-27T15:39:20.0120627Z Entering 'third_party/tensorpipe'
2022-09-27T15:39:20.0163731Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T15:39:20.0206501Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T15:39:20.0248081Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T15:39:20.0290250Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T15:39:20.0331740Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T15:39:20.0376495Z Entering 'third_party/zstd'
2022-09-27T15:39:20.0428282Z ##[endgroup]
2022-09-27T15:39:20.0474449Z [command]/usr/bin/git log -1 --format='%H'
2022-09-27T15:39:20.0503266Z '52424e2bf38e454d535881fed9628d3e20f4f944'
2022-09-27T15:39:20.0654840Z Prepare all required actions
2022-09-27T15:39:20.0728684Z ##[group]Run ./.github/actions/setup-linux
2022-09-27T15:39:20.0728976Z env:
2022-09-27T15:39:20.0729222Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:20.0729471Z ##[endgroup]
2022-09-27T15:39:20.0749756Z ##[group]Run set -euo pipefail
2022-09-27T15:39:20.0750077Z set -euo pipefail
2022-09-27T15:39:20.0750375Z function get_ec2_metadata() {
2022-09-27T15:39:20.0750708Z   # Pulled from instance metadata endpoint for EC2
2022-09-27T15:39:20.0751203Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-09-27T15:39:20.0751615Z   category=$1
2022-09-27T15:39:20.0751958Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-09-27T15:39:20.0752265Z }
2022-09-27T15:39:20.0752689Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-09-27T15:39:20.0753086Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-09-27T15:39:20.0753467Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-09-27T15:39:20.0753819Z echo "system info $(uname -a)"
2022-09-27T15:39:20.0766757Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:20.0767036Z env:
2022-09-27T15:39:20.0767271Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:20.0767523Z ##[endgroup]
2022-09-27T15:39:20.0867613Z ami-id: ami-096198a0bccc6bad4
2022-09-27T15:39:20.0930398Z instance-id: i-037610acd6d6b81a9
2022-09-27T15:39:20.0992912Z instance-type: g3.8xlarge
2022-09-27T15:39:20.1001683Z system info Linux ip-10-0-4-203.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-09-27T15:39:20.1022752Z ##[group]Run if systemctl is-active --quiet docker; then
2022-09-27T15:39:20.1023131Z if systemctl is-active --quiet docker; then
2022-09-27T15:39:20.1023471Z     echo "Docker daemon is running...";
2022-09-27T15:39:20.1023728Z else
2022-09-27T15:39:20.1024046Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-09-27T15:39:20.1024351Z fi
2022-09-27T15:39:20.1035764Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:20.1036055Z env:
2022-09-27T15:39:20.1036285Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:20.1036522Z ##[endgroup]
2022-09-27T15:39:20.1087932Z Docker daemon is running...
2022-09-27T15:39:20.1107093Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-09-27T15:39:20.1107562Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-09-27T15:39:20.1107942Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:39:20.1108493Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-09-27T15:39:20.1108959Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-09-27T15:39:20.1119777Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:20.1120070Z env:
2022-09-27T15:39:20.1120664Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:20.1120942Z   AWS_RETRY_MODE: standard
2022-09-27T15:39:20.1121200Z   AWS_MAX_ATTEMPTS: 5
2022-09-27T15:39:20.1121453Z   AWS_DEFAULT_REGION: us-east-1
2022-09-27T15:39:20.1121711Z ##[endgroup]
2022-09-27T15:39:21.0446219Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-09-27T15:39:21.0446722Z Configure a credential helper to remove this warning. See
2022-09-27T15:39:21.0447259Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-09-27T15:39:21.0447511Z 
2022-09-27T15:39:21.0448293Z Login Succeeded
2022-09-27T15:39:21.0531042Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:39:21.0531464Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:39:21.0531939Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-09-27T15:39:21.0544421Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:21.0544718Z env:
2022-09-27T15:39:21.0544958Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:21.0545200Z ##[endgroup]
2022-09-27T15:39:21.0632885Z Prepare all required actions
2022-09-27T15:39:21.0633563Z Getting action download info
2022-09-27T15:39:21.2299363Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:105f7619adc4054f5f1be5f79ebd354d82384638)
2022-09-27T15:39:21.3564030Z ##[group]Run ./.github/actions/setup-ssh
2022-09-27T15:39:21.3564304Z with:
2022-09-27T15:39:21.3564753Z   github-secret: ***
2022-09-27T15:39:21.3565002Z env:
2022-09-27T15:39:21.3565241Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:21.3565480Z ##[endgroup]
2022-09-27T15:39:21.3591712Z ##[group]Run seemethere/add-github-ssh-key@v1
2022-09-27T15:39:21.3592105Z with:
2022-09-27T15:39:21.3592493Z   GITHUB_TOKEN: ***
2022-09-27T15:39:21.3592749Z   activate-with-label: false
2022-09-27T15:39:21.3593020Z   label: with-ssh
2022-09-27T15:39:21.3593285Z   remove-existing-keys: true
2022-09-27T15:39:21.3593519Z env:
2022-09-27T15:39:21.3593757Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:21.3594014Z ##[endgroup]
2022-09-27T15:39:21.8849289Z Grabbing public ssh keys from https://github.com/kongzii.keys
2022-09-27T15:39:21.9725048Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2022-09-27T15:39:21.9745004Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2022-09-27T15:39:21.9780304Z Login using: ssh ec2-user@ec2-34-224-94-146.compute-1.amazonaws.com
2022-09-27T15:39:21.9837726Z Prepare all required actions
2022-09-27T15:39:21.9862138Z ##[group]Run ./.github/actions/pull-docker-image
2022-09-27T15:39:21.9862429Z with:
2022-09-27T15:39:21.9862920Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:39:21.9863389Z env:
2022-09-27T15:39:21.9863634Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:21.9863894Z ##[endgroup]
2022-09-27T15:39:21.9881516Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:39:21.9881885Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-09-27T15:39:21.9882252Z # ignore output since only exit code is used for conditional
2022-09-27T15:39:21.9882621Z # only pull docker image if it's not available locally
2022-09-27T15:39:21.9883025Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-09-27T15:39:21.9883436Z   retry docker pull "${DOCKER_IMAGE}"
2022-09-27T15:39:21.9883696Z fi
2022-09-27T15:39:21.9896093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:39:21.9896392Z env:
2022-09-27T15:39:21.9896638Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:39:21.9897142Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:39:21.9897629Z ##[endgroup]
2022-09-27T15:39:22.2294733Z e66cf5fa0a4d4ed512901b12ccdab95cca946a29: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2022-09-27T15:39:22.2295433Z 40dd5be53814: Pulling fs layer
2022-09-27T15:39:22.2295709Z bd44602516a4: Pulling fs layer
2022-09-27T15:39:22.2296117Z 8ebfb31ea67d: Pulling fs layer
2022-09-27T15:39:22.2296613Z 1589dc294916: Pulling fs layer
2022-09-27T15:39:22.2296885Z 2c3a764ff1ef: Pulling fs layer
2022-09-27T15:39:22.2297159Z 2fb24fb5f7cb: Pulling fs layer
2022-09-27T15:39:22.2297454Z d6e4b45751c9: Pulling fs layer
2022-09-27T15:39:22.2297705Z 98a26bc0781e: Pulling fs layer
2022-09-27T15:39:22.2297971Z 07c42b0591b2: Pulling fs layer
2022-09-27T15:39:22.2298239Z 9be88323b57e: Pulling fs layer
2022-09-27T15:39:22.2298487Z 2c3a764ff1ef: Waiting
2022-09-27T15:39:22.2298750Z 2c7b68ade49f: Pulling fs layer
2022-09-27T15:39:22.2299009Z 2fb24fb5f7cb: Waiting
2022-09-27T15:39:22.2299254Z 44206692de1d: Pulling fs layer
2022-09-27T15:39:22.2299519Z f751461554fa: Pulling fs layer
2022-09-27T15:39:22.2299805Z 316750fef2e6: Pulling fs layer
2022-09-27T15:39:22.2300045Z 98a26bc0781e: Waiting
2022-09-27T15:39:22.2300287Z d6e4b45751c9: Waiting
2022-09-27T15:39:22.2300527Z f751461554fa: Waiting
2022-09-27T15:39:22.2300762Z c069021d810b: Pulling fs layer
2022-09-27T15:39:22.2301036Z e0fdd58e805b: Pulling fs layer
2022-09-27T15:39:22.2301304Z 751286b45698: Pulling fs layer
2022-09-27T15:39:22.2301550Z 0c8bd29be614: Pulling fs layer
2022-09-27T15:39:22.2301806Z 44206692de1d: Waiting
2022-09-27T15:39:22.2302048Z 07c42b0591b2: Waiting
2022-09-27T15:39:22.2302297Z 3bb9e7ea569e: Pulling fs layer
2022-09-27T15:39:22.2302533Z c069021d810b: Waiting
2022-09-27T15:39:22.2302790Z efeff9c74fbf: Pulling fs layer
2022-09-27T15:39:22.2303057Z 81a5271d43c8: Pulling fs layer
2022-09-27T15:39:22.2303496Z e0fdd58e805b: Waiting
2022-09-27T15:39:22.2303749Z 903ca36d4d71: Pulling fs layer
2022-09-27T15:39:22.2304003Z 3bb9e7ea569e: Waiting
2022-09-27T15:39:22.2304241Z d52c758f8e75: Pulling fs layer
2022-09-27T15:39:22.2304500Z efeff9c74fbf: Waiting
2022-09-27T15:39:22.2304761Z a4ce2fdd9133: Pulling fs layer
2022-09-27T15:39:22.2305003Z 9be88323b57e: Waiting
2022-09-27T15:39:22.2305239Z 903ca36d4d71: Waiting
2022-09-27T15:39:22.2305494Z cae8823a1cd1: Pulling fs layer
2022-09-27T15:39:22.2305734Z d52c758f8e75: Waiting
2022-09-27T15:39:22.2305982Z cae8823a1cd1: Waiting
2022-09-27T15:39:22.2306229Z 3298fe919163: Pulling fs layer
2022-09-27T15:39:22.2306497Z 2c7b68ade49f: Waiting
2022-09-27T15:39:22.2306848Z b9b9b9d06eef: Pulling fs layer
2022-09-27T15:39:22.2307109Z 3298fe919163: Waiting
2022-09-27T15:39:22.2307347Z 62fa99d47769: Pulling fs layer
2022-09-27T15:39:22.2307611Z 17acc9e30503: Pulling fs layer
2022-09-27T15:39:22.2307873Z e8b4222e7a59: Pulling fs layer
2022-09-27T15:39:22.2308147Z b752992950f8: Pulling fs layer
2022-09-27T15:39:22.2308391Z cc8443c330a0: Pulling fs layer
2022-09-27T15:39:22.2308659Z c2fcfa2400df: Pulling fs layer
2022-09-27T15:39:22.2308915Z 0c8bd29be614: Waiting
2022-09-27T15:39:22.2309136Z e8b4222e7a59: Waiting
2022-09-27T15:39:22.2309390Z dce607cbd09e: Pulling fs layer
2022-09-27T15:39:22.2309646Z a4ce2fdd9133: Waiting
2022-09-27T15:39:22.2309871Z b9b9b9d06eef: Waiting
2022-09-27T15:39:22.2310107Z b752992950f8: Waiting
2022-09-27T15:39:22.2310349Z 45b253446018: Pulling fs layer
2022-09-27T15:39:22.2310592Z 752f98c7a6d7: Pulling fs layer
2022-09-27T15:39:22.2310853Z 410e31c94a04: Pulling fs layer
2022-09-27T15:39:22.2311118Z aa4bb3ec24a7: Pulling fs layer
2022-09-27T15:39:22.2311376Z 6a9eea4b3aa4: Pulling fs layer
2022-09-27T15:39:22.2311644Z a823f5718e87: Pulling fs layer
2022-09-27T15:39:22.2311911Z 788ace045743: Pulling fs layer
2022-09-27T15:39:22.2312138Z 45b253446018: Waiting
2022-09-27T15:39:22.2312389Z c76c6ad2ac0f: Pulling fs layer
2022-09-27T15:39:22.2312665Z 6cd5f9a2c4ae: Pulling fs layer
2022-09-27T15:39:22.2312920Z 577da355ab1b: Pulling fs layer
2022-09-27T15:39:22.2313175Z dce607cbd09e: Waiting
2022-09-27T15:39:22.2313435Z 649c4428b346: Pulling fs layer
2022-09-27T15:39:22.2313666Z aa4bb3ec24a7: Waiting
2022-09-27T15:39:22.2313908Z 577da355ab1b: Waiting
2022-09-27T15:39:22.2314144Z 788ace045743: Waiting
2022-09-27T15:39:22.2314380Z eadc05ea2cd3: Pulling fs layer
2022-09-27T15:39:22.2314638Z c76c6ad2ac0f: Waiting
2022-09-27T15:39:22.2314887Z 93e5a7080833: Pulling fs layer
2022-09-27T15:39:22.2315134Z e6d72a41a09b: Pulling fs layer
2022-09-27T15:39:22.2315387Z eadc05ea2cd3: Waiting
2022-09-27T15:39:22.2315642Z acb01049a64b: Pulling fs layer
2022-09-27T15:39:22.2315886Z 6a9eea4b3aa4: Waiting
2022-09-27T15:39:22.2316142Z 343cc73c5973: Pulling fs layer
2022-09-27T15:39:22.2316400Z 93e5a7080833: Waiting
2022-09-27T15:39:22.2316649Z 7d69e17e7339: Pulling fs layer
2022-09-27T15:39:22.2316898Z 5d0b32cc6f2a: Pulling fs layer
2022-09-27T15:39:22.2317151Z 649c4428b346: Waiting
2022-09-27T15:39:22.2317405Z a6c12031bfcf: Pulling fs layer
2022-09-27T15:39:22.2317640Z 7d69e17e7339: Waiting
2022-09-27T15:39:22.2317875Z acb01049a64b: Waiting
2022-09-27T15:39:22.2318112Z 5d0b32cc6f2a: Waiting
2022-09-27T15:39:22.2318331Z e6d72a41a09b: Waiting
2022-09-27T15:39:22.2318570Z a6c12031bfcf: Waiting
2022-09-27T15:39:22.2318804Z 410e31c94a04: Waiting
2022-09-27T15:39:22.3618860Z bd44602516a4: Verifying Checksum
2022-09-27T15:39:22.3619188Z bd44602516a4: Download complete
2022-09-27T15:39:22.4429113Z 1589dc294916: Download complete
2022-09-27T15:39:22.5419868Z 8ebfb31ea67d: Verifying Checksum
2022-09-27T15:39:22.5420197Z 8ebfb31ea67d: Download complete
2022-09-27T15:39:22.5562097Z 40dd5be53814: Verifying Checksum
2022-09-27T15:39:22.5562419Z 40dd5be53814: Download complete
2022-09-27T15:39:22.5868390Z 2c3a764ff1ef: Verifying Checksum
2022-09-27T15:39:22.5868686Z 2c3a764ff1ef: Download complete
2022-09-27T15:39:22.6510398Z d6e4b45751c9: Verifying Checksum
2022-09-27T15:39:22.6511065Z d6e4b45751c9: Download complete
2022-09-27T15:39:22.7330858Z 07c42b0591b2: Verifying Checksum
2022-09-27T15:39:22.7331215Z 07c42b0591b2: Download complete
2022-09-27T15:39:22.8131175Z 9be88323b57e: Download complete
2022-09-27T15:39:23.3121858Z 40dd5be53814: Pull complete
2022-09-27T15:39:23.5891960Z bd44602516a4: Pull complete
2022-09-27T15:39:24.1278733Z 8ebfb31ea67d: Pull complete
2022-09-27T15:39:24.2649175Z 1589dc294916: Pull complete
2022-09-27T15:39:24.3897845Z 2c3a764ff1ef: Pull complete
2022-09-27T15:39:24.9417861Z 2c7b68ade49f: Verifying Checksum
2022-09-27T15:39:24.9418230Z 2c7b68ade49f: Download complete
2022-09-27T15:39:25.0186050Z 44206692de1d: Verifying Checksum
2022-09-27T15:39:25.0186379Z 44206692de1d: Download complete
2022-09-27T15:39:25.0909800Z f751461554fa: Verifying Checksum
2022-09-27T15:39:25.0910513Z f751461554fa: Download complete
2022-09-27T15:39:25.1729046Z 316750fef2e6: Verifying Checksum
2022-09-27T15:39:25.1729348Z 316750fef2e6: Download complete
2022-09-27T15:39:25.9367682Z c069021d810b: Verifying Checksum
2022-09-27T15:39:25.9368052Z c069021d810b: Download complete
2022-09-27T15:39:26.0302788Z e0fdd58e805b: Verifying Checksum
2022-09-27T15:39:26.0303116Z e0fdd58e805b: Download complete
2022-09-27T15:39:26.1216442Z 751286b45698: Verifying Checksum
2022-09-27T15:39:26.1216781Z 751286b45698: Download complete
2022-09-27T15:39:33.7835269Z 2fb24fb5f7cb: Verifying Checksum
2022-09-27T15:39:33.7835926Z 2fb24fb5f7cb: Download complete
2022-09-27T15:39:33.8695545Z 3bb9e7ea569e: Verifying Checksum
2022-09-27T15:39:33.8696229Z 3bb9e7ea569e: Download complete
2022-09-27T15:39:33.9479758Z efeff9c74fbf: Verifying Checksum
2022-09-27T15:39:33.9480839Z efeff9c74fbf: Download complete
2022-09-27T15:39:34.0286016Z 81a5271d43c8: Download complete
2022-09-27T15:39:34.1046035Z 903ca36d4d71: Verifying Checksum
2022-09-27T15:39:34.1046619Z 903ca36d4d71: Download complete
2022-09-27T15:39:34.2001025Z d52c758f8e75: Verifying Checksum
2022-09-27T15:39:34.2001579Z d52c758f8e75: Download complete
2022-09-27T15:39:34.2846187Z a4ce2fdd9133: Download complete
2022-09-27T15:39:35.2279604Z cae8823a1cd1: Verifying Checksum
2022-09-27T15:39:35.2279962Z cae8823a1cd1: Download complete
2022-09-27T15:39:35.3112569Z 3298fe919163: Verifying Checksum
2022-09-27T15:39:35.3113198Z 3298fe919163: Download complete
2022-09-27T15:39:35.3917939Z b9b9b9d06eef: Verifying Checksum
2022-09-27T15:39:35.3918575Z b9b9b9d06eef: Download complete
2022-09-27T15:39:35.5012401Z 62fa99d47769: Verifying Checksum
2022-09-27T15:39:35.5012803Z 62fa99d47769: Download complete
2022-09-27T15:39:35.5882688Z 17acc9e30503: Download complete
2022-09-27T15:39:35.6663495Z e8b4222e7a59: Download complete
2022-09-27T15:39:36.9508356Z 98a26bc0781e: Verifying Checksum
2022-09-27T15:39:36.9509001Z 98a26bc0781e: Download complete
2022-09-27T15:39:37.0289907Z cc8443c330a0: Verifying Checksum
2022-09-27T15:39:37.0290533Z cc8443c330a0: Download complete
2022-09-27T15:39:37.1205017Z c2fcfa2400df: Verifying Checksum
2022-09-27T15:39:37.1205347Z c2fcfa2400df: Download complete
2022-09-27T15:39:37.4564223Z dce607cbd09e: Download complete
2022-09-27T15:39:37.5294458Z 45b253446018: Verifying Checksum
2022-09-27T15:39:37.6025562Z 752f98c7a6d7: Verifying Checksum
2022-09-27T15:39:37.6026131Z 752f98c7a6d7: Download complete
2022-09-27T15:39:37.6598023Z b752992950f8: Verifying Checksum
2022-09-27T15:39:37.6598699Z b752992950f8: Download complete
2022-09-27T15:39:37.7259505Z aa4bb3ec24a7: Download complete
2022-09-27T15:39:37.8655902Z 410e31c94a04: Verifying Checksum
2022-09-27T15:39:37.8656516Z 410e31c94a04: Download complete
2022-09-27T15:39:37.9565456Z a823f5718e87: Verifying Checksum
2022-09-27T15:39:37.9566086Z a823f5718e87: Download complete
2022-09-27T15:39:38.0695865Z 788ace045743: Verifying Checksum
2022-09-27T15:39:38.0696483Z 788ace045743: Download complete
2022-09-27T15:39:38.1789049Z 6a9eea4b3aa4: Verifying Checksum
2022-09-27T15:39:38.1789467Z 6a9eea4b3aa4: Download complete
2022-09-27T15:39:38.2776103Z 6cd5f9a2c4ae: Download complete
2022-09-27T15:39:38.4019806Z 577da355ab1b: Verifying Checksum
2022-09-27T15:39:38.4020438Z 577da355ab1b: Download complete
2022-09-27T15:39:38.4847786Z 649c4428b346: Verifying Checksum
2022-09-27T15:39:38.4848128Z 649c4428b346: Download complete
2022-09-27T15:39:38.5539079Z eadc05ea2cd3: Verifying Checksum
2022-09-27T15:39:38.5539723Z eadc05ea2cd3: Download complete
2022-09-27T15:39:38.7500496Z 93e5a7080833: Verifying Checksum
2022-09-27T15:39:38.8628159Z e6d72a41a09b: Download complete
2022-09-27T15:39:39.4738160Z acb01049a64b: Verifying Checksum
2022-09-27T15:39:39.4738883Z acb01049a64b: Download complete
2022-09-27T15:39:39.5447860Z 343cc73c5973: Verifying Checksum
2022-09-27T15:39:39.5448511Z 343cc73c5973: Download complete
2022-09-27T15:39:40.8360025Z c76c6ad2ac0f: Verifying Checksum
2022-09-27T15:39:40.8360917Z c76c6ad2ac0f: Download complete
2022-09-27T15:39:40.9329481Z 5d0b32cc6f2a: Verifying Checksum
2022-09-27T15:39:40.9329840Z 5d0b32cc6f2a: Download complete
2022-09-27T15:39:41.0134185Z a6c12031bfcf: Verifying Checksum
2022-09-27T15:39:41.0134573Z a6c12031bfcf: Download complete
2022-09-27T15:39:47.5358204Z 2fb24fb5f7cb: Pull complete
2022-09-27T15:39:47.6617980Z d6e4b45751c9: Pull complete
2022-09-27T15:40:08.9389398Z 98a26bc0781e: Pull complete
2022-09-27T15:40:10.8175812Z 07c42b0591b2: Pull complete
2022-09-27T15:40:12.6924683Z 9be88323b57e: Pull complete
2022-09-27T15:40:17.8187342Z 0c8bd29be614: Verifying Checksum
2022-09-27T15:40:17.8187701Z 0c8bd29be614: Download complete
2022-09-27T15:40:20.7828313Z 2c7b68ade49f: Pull complete
2022-09-27T15:40:22.6623903Z 44206692de1d: Pull complete
2022-09-27T15:40:24.5344768Z f751461554fa: Pull complete
2022-09-27T15:40:26.5412704Z 316750fef2e6: Pull complete
2022-09-27T15:40:30.9311667Z c069021d810b: Pull complete
2022-09-27T15:40:33.1907580Z e0fdd58e805b: Pull complete
2022-09-27T15:40:35.0692951Z 751286b45698: Pull complete
2022-09-27T15:40:39.3389104Z 7d69e17e7339: Verifying Checksum
2022-09-27T15:40:39.3389486Z 7d69e17e7339: Download complete
2022-09-27T15:41:12.5741895Z 0c8bd29be614: Pull complete
2022-09-27T15:41:14.4817892Z 3bb9e7ea569e: Pull complete
2022-09-27T15:41:16.3791736Z efeff9c74fbf: Pull complete
2022-09-27T15:41:18.3880079Z 81a5271d43c8: Pull complete
2022-09-27T15:41:20.2463430Z 903ca36d4d71: Pull complete
2022-09-27T15:41:22.0907826Z d52c758f8e75: Pull complete
2022-09-27T15:41:23.9351733Z a4ce2fdd9133: Pull complete
2022-09-27T15:41:28.2690761Z cae8823a1cd1: Pull complete
2022-09-27T15:41:30.3549629Z 3298fe919163: Pull complete
2022-09-27T15:41:32.5325000Z b9b9b9d06eef: Pull complete
2022-09-27T15:41:35.7114846Z 62fa99d47769: Pull complete
2022-09-27T15:41:38.1558910Z 17acc9e30503: Pull complete
2022-09-27T15:41:40.3636784Z e8b4222e7a59: Pull complete
2022-09-27T15:41:48.8173221Z b752992950f8: Pull complete
2022-09-27T15:41:50.6643317Z cc8443c330a0: Pull complete
2022-09-27T15:41:52.4919479Z c2fcfa2400df: Pull complete
2022-09-27T15:41:55.1538284Z dce607cbd09e: Pull complete
2022-09-27T15:41:57.1077095Z 45b253446018: Pull complete
2022-09-27T15:41:58.8738325Z 752f98c7a6d7: Pull complete
2022-09-27T15:42:01.0948313Z 410e31c94a04: Pull complete
2022-09-27T15:42:02.9410520Z aa4bb3ec24a7: Pull complete
2022-09-27T15:42:06.6583249Z 6a9eea4b3aa4: Pull complete
2022-09-27T15:42:07.7545339Z a823f5718e87: Pull complete
2022-09-27T15:42:07.8603935Z 788ace045743: Pull complete
2022-09-27T15:42:12.3781934Z c76c6ad2ac0f: Pull complete
2022-09-27T15:42:12.5014534Z 6cd5f9a2c4ae: Pull complete
2022-09-27T15:42:12.6212836Z 577da355ab1b: Pull complete
2022-09-27T15:42:12.7219372Z 649c4428b346: Pull complete
2022-09-27T15:42:13.3371727Z eadc05ea2cd3: Pull complete
2022-09-27T15:42:16.5660641Z 93e5a7080833: Pull complete
2022-09-27T15:42:16.6933489Z e6d72a41a09b: Pull complete
2022-09-27T15:42:18.6070402Z acb01049a64b: Pull complete
2022-09-27T15:42:18.7334804Z 343cc73c5973: Pull complete
2022-09-27T15:43:00.3295991Z 7d69e17e7339: Pull complete
2022-09-27T15:43:02.2769376Z 5d0b32cc6f2a: Pull complete
2022-09-27T15:43:04.1255572Z a6c12031bfcf: Pull complete
2022-09-27T15:43:05.4728666Z Digest: sha256:9bb261bc4d8aeb82a71b1f0709da9c979e85a12a79c4a85c3fe3adddddcb2663
2022-09-27T15:43:05.9733202Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:43:06.2651324Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:43:06.2741479Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-09-27T15:43:06.2741959Z with:
2022-09-27T15:43:06.2742281Z   timeout_minutes: 10
2022-09-27T15:43:06.2742552Z   max_attempts: 3
2022-09-27T15:43:06.2743021Z   command: set -ex
bash .github/scripts/install_nvidia_utils_linux.sh
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-09-27T15:43:06.2743476Z   retry_wait_seconds: 10
2022-09-27T15:43:06.2743805Z   polling_interval_seconds: 1
2022-09-27T15:43:06.2744148Z   warning_on_retry: true
2022-09-27T15:43:06.2744498Z   continue_on_error: false
2022-09-27T15:43:06.2744757Z env:
2022-09-27T15:43:06.2745063Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:43:06.2745423Z ##[endgroup]
2022-09-27T15:43:06.3292163Z 
2022-09-27T15:43:06.3354820Z + bash .github/scripts/install_nvidia_utils_linux.sh
2022-09-27T15:43:06.3357139Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.57.run ==
2022-09-27T15:43:06.3358123Z + HAS_NVIDIA_DRIVER=0
2022-09-27T15:43:06.3361286Z ++ command -v nvidia-smi
2022-09-27T15:43:06.3363293Z + '[' -x '' ']'
2022-09-27T15:43:06.3363855Z + '[' 0 -eq 0 ']'
2022-09-27T15:43:06.3364208Z + sudo yum groupinstall -y 'Development Tools'
2022-09-27T15:43:06.8069905Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:43:07.1085852Z Resolving Dependencies
2022-09-27T15:43:07.1090936Z --> Running transaction check
2022-09-27T15:43:07.1093795Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-09-27T15:43:07.1310807Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-09-27T15:43:07.3376232Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-09-27T15:43:07.3378001Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-09-27T15:43:07.3424386Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-09-27T15:43:07.3431284Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-09-27T15:43:07.3442285Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-09-27T15:43:07.3554300Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-09-27T15:43:07.3561504Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-09-27T15:43:07.3605790Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64
2022-09-27T15:43:07.3629970Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-09-27T15:43:07.3638687Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-09-27T15:43:07.3646835Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-09-27T15:43:07.3740669Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-09-27T15:43:07.3880409Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-09-27T15:43:07.3899599Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.4069126Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4089433Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4144588Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4196818Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4253176Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4305919Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4360272Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4415229Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4563047Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4583808Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4605139Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.4632224Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.4666101Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64
2022-09-27T15:43:07.4729573Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-09-27T15:43:07.4744045Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-09-27T15:43:07.4796800Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-09-27T15:43:07.4811269Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-09-27T15:43:07.4830200Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-09-27T15:43:07.4861363Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-09-27T15:43:07.4897639Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-09-27T15:43:07.4923200Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-09-27T15:43:07.4956224Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-09-27T15:43:07.5199921Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-09-27T15:43:07.5219466Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-09-27T15:43:07.5242763Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-09-27T15:43:07.5414170Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5433927Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5434865Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5436065Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5436764Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5437408Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5438048Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5438685Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5439315Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5439951Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5440830Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5441457Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5442265Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5442894Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5443846Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5444467Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5462721Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5483872Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-09-27T15:43:07.5507981Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-09-27T15:43:07.5530943Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed
2022-09-27T15:43:07.5577971Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-09-27T15:43:07.5595905Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-09-27T15:43:07.5608726Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-09-27T15:43:07.5772720Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-09-27T15:43:07.5785876Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:43:07.5800799Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:43:07.5814816Z --> Running transaction check
2022-09-27T15:43:07.5818226Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-09-27T15:43:07.5893016Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-09-27T15:43:07.5931359Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-09-27T15:43:07.5945712Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.6021049Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-09-27T15:43:07.6031552Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
2022-09-27T15:43:07.6032576Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-09-27T15:43:07.6104139Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-09-27T15:43:07.6170663Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-09-27T15:43:07.6180032Z ---> Package glibc-devel.x86_64 0:2.26-60.amzn2 will be installed
2022-09-27T15:43:07.6296996Z --> Processing Dependency: glibc-headers = 2.26-60.amzn2 for package: glibc-devel-2.26-60.amzn2.x86_64
2022-09-27T15:43:07.6323570Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-60.amzn2.x86_64
2022-09-27T15:43:07.6324381Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1 will be installed
2022-09-27T15:43:07.6329450Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.6342674Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.6369915Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.6405010Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.6420857Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-09-27T15:43:07.6433036Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.6448136Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.6472913Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed
2022-09-27T15:43:07.6518127Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-09-27T15:43:07.6533431Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-09-27T15:43:07.6554120Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-09-27T15:43:07.6630153Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:43:07.6667926Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:43:07.6668953Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:43:07.6688813Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:43:07.6706357Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-09-27T15:43:07.6712601Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-09-27T15:43:07.6762928Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-09-27T15:43:07.6857778Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-09-27T15:43:07.6870322Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-09-27T15:43:07.6885737Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-09-27T15:43:07.6886877Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-09-27T15:43:07.6915105Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-09-27T15:43:07.7118878Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:43:07.7133586Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:43:07.7159228Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:43:07.7159810Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-09-27T15:43:07.7284167Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64
2022-09-27T15:43:07.8278164Z --> Running transaction check
2022-09-27T15:43:07.8278655Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-09-27T15:43:07.8289308Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-09-27T15:43:07.8315446Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-09-27T15:43:07.8316566Z ---> Package glibc-headers.x86_64 0:2.26-60.amzn2 will be installed
2022-09-27T15:43:07.8389963Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-60.amzn2.x86_64
2022-09-27T15:43:07.9459492Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-60.amzn2.x86_64
2022-09-27T15:43:07.9460037Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-09-27T15:43:07.9527312Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-09-27T15:43:07.9553950Z ---> Package kernel-devel.x86_64 0:4.14.291-218.527.amzn2 will be installed
2022-09-27T15:43:07.9581448Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.291-218.527.amzn2.x86_64
2022-09-27T15:43:07.9601187Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-09-27T15:43:07.9629491Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-09-27T15:43:07.9647858Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-09-27T15:43:07.9696759Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-09-27T15:43:07.9717264Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-09-27T15:43:07.9718321Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-09-27T15:43:07.9719140Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-09-27T15:43:07.9733091Z --> Running transaction check
2022-09-27T15:43:07.9733727Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-09-27T15:43:07.9751133Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-09-27T15:43:07.9763442Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-09-27T15:43:07.9787070Z ---> Package kernel-headers.x86_64 0:4.14.291-218.527.amzn2 will be installed
2022-09-27T15:43:07.9788150Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-09-27T15:43:07.9806112Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-09-27T15:43:07.9863417Z --> Running transaction check
2022-09-27T15:43:07.9864054Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1 will be installed
2022-09-27T15:43:08.2504721Z --> Finished Dependency Resolution
2022-09-27T15:43:08.3276968Z 
2022-09-27T15:43:08.3277356Z Dependencies Resolved
2022-09-27T15:43:08.3390404Z 
2022-09-27T15:43:08.3390784Z ================================================================================
2022-09-27T15:43:08.3391172Z  Package                 Arch     Version                    Repository    Size
2022-09-27T15:43:08.3391527Z ================================================================================
2022-09-27T15:43:08.3396636Z Installing for group install "Development Tools":
2022-09-27T15:43:08.3397265Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-09-27T15:43:08.3397727Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-09-27T15:43:08.3398188Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-09-27T15:43:08.3398632Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-09-27T15:43:08.3399058Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-09-27T15:43:08.3399496Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-09-27T15:43:08.3399930Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-09-27T15:43:08.3400984Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-09-27T15:43:08.3401435Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-09-27T15:43:08.3401878Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-09-27T15:43:08.3402306Z  gcc                     x86_64   7.3.1-15.amzn2             amzn2-core    22 M
2022-09-27T15:43:08.3402719Z  gcc-c++                 x86_64   7.3.1-15.amzn2             amzn2-core    13 M
2022-09-27T15:43:08.3403169Z  gcc-gfortran            x86_64   7.3.1-15.amzn2             amzn2-core    11 M
2022-09-27T15:43:08.3403615Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-09-27T15:43:08.3404050Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-09-27T15:43:08.3404468Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-09-27T15:43:08.3404904Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-09-27T15:43:08.3409441Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-09-27T15:43:08.3410291Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-09-27T15:43:08.3411136Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-09-27T15:43:08.3411836Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-09-27T15:43:08.3412304Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-09-27T15:43:08.3412725Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-09-27T15:43:08.3413183Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.14        amzn2-core    90 k
2022-09-27T15:43:08.3413650Z  systemtap               x86_64   4.5-1.amzn2.0.1            amzn2-core    12 k
2022-09-27T15:43:08.3413959Z Installing for dependencies:
2022-09-27T15:43:08.3414377Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-09-27T15:43:08.3414814Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-09-27T15:43:08.3415470Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-09-27T15:43:08.3415925Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-09-27T15:43:08.3416361Z  cpp                     x86_64   7.3.1-15.amzn2             amzn2-core   9.2 M
2022-09-27T15:43:08.3416880Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-09-27T15:43:08.3417313Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-09-27T15:43:08.3417774Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-09-27T15:43:08.3418251Z  emacs-filesystem        noarch   1:27.2-4.amzn2.0.1         amzn2-core    67 k
2022-09-27T15:43:08.3418683Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-09-27T15:43:08.3419142Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-09-27T15:43:08.3419618Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-09-27T15:43:08.3420071Z  glibc-devel             x86_64   2.26-60.amzn2              amzn2-core   994 k
2022-09-27T15:43:08.3420500Z  glibc-headers           x86_64   2.26-60.amzn2              amzn2-core   515 k
2022-09-27T15:43:08.3420946Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-09-27T15:43:08.3421403Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.1        amzn2-core    23 k
2022-09-27T15:43:08.3421844Z  kernel-devel            x86_64   4.14.291-218.527.amzn2     amzn2-core    13 M
2022-09-27T15:43:08.3422300Z  kernel-headers          x86_64   4.14.291-218.527.amzn2     amzn2-core   1.2 M
2022-09-27T15:43:08.3422736Z  libatomic               x86_64   7.3.1-15.amzn2             amzn2-core    46 k
2022-09-27T15:43:08.3423174Z  libcilkrts              x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-09-27T15:43:08.3423600Z  libgfortran             x86_64   7.3.1-15.amzn2             amzn2-core   536 k
2022-09-27T15:43:08.3424027Z  libitm                  x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-09-27T15:43:08.3424465Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-09-27T15:43:08.3424884Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-09-27T15:43:08.3425322Z  libmpx                  x86_64   7.3.1-15.amzn2             amzn2-core    51 k
2022-09-27T15:43:08.3425751Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-09-27T15:43:08.3426187Z  libquadmath             x86_64   7.3.1-15.amzn2             amzn2-core   189 k
2022-09-27T15:43:08.3426609Z  libsanitizer            x86_64   7.3.1-15.amzn2             amzn2-core   642 k
2022-09-27T15:43:08.3427039Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-09-27T15:43:08.3427462Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-09-27T15:43:08.3427882Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-09-27T15:43:08.3428307Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-09-27T15:43:08.3428737Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-09-27T15:43:08.3429197Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-09-27T15:43:08.3429656Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-09-27T15:43:08.3430151Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-09-27T15:43:08.3430634Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-09-27T15:43:08.3431107Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-09-27T15:43:08.3431563Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-09-27T15:43:08.3432110Z  systemtap-client        x86_64   4.5-1.amzn2.0.1            amzn2-core   3.7 M
2022-09-27T15:43:08.3432571Z  systemtap-devel         x86_64   4.5-1.amzn2.0.1            amzn2-core   2.3 M
2022-09-27T15:43:08.3433001Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-09-27T15:43:08.3433439Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.1         amzn2-core    50 k
2022-09-27T15:43:08.3433715Z 
2022-09-27T15:43:08.3433838Z Transaction Summary
2022-09-27T15:43:08.3434131Z ================================================================================
2022-09-27T15:43:08.3434436Z Install  25 Packages (+43 Dependent packages)
2022-09-27T15:43:08.3434638Z 
2022-09-27T15:43:08.3434762Z Total download size: 96 M
2022-09-27T15:43:08.3435040Z Installed size: 303 M
2022-09-27T15:43:08.3435290Z Downloading packages:
2022-09-27T15:43:08.3441689Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-09-27T15:43:09.7272438Z --------------------------------------------------------------------------------
2022-09-27T15:43:09.7272932Z Total                                               69 MB/s |  96 MB  00:01     
2022-09-27T15:43:09.8338423Z Running transaction check
2022-09-27T15:43:09.9110568Z Running transaction test
2022-09-27T15:43:12.3534140Z Transaction test succeeded
2022-09-27T15:43:12.3537439Z Running transaction
2022-09-27T15:43:17.8148452Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/68 
2022-09-27T15:43:20.3067217Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/68 
2022-09-27T15:43:22.8036920Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/68 
2022-09-27T15:43:25.2676127Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/68 
2022-09-27T15:43:27.7218991Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/68 
2022-09-27T15:43:30.0967539Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/68 
2022-09-27T15:43:32.4109299Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/68 
2022-09-27T15:43:34.8794367Z   Installing : libquadmath-7.3.1-15.amzn2.x86_64                           8/68 
2022-09-27T15:43:35.0907384Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/68 
2022-09-27T15:43:35.1728486Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/68 
2022-09-27T15:43:36.2343337Z   Installing : libgfortran-7.3.1-15.amzn2.x86_64                          11/68 
2022-09-27T15:43:36.2728619Z   Installing : cpp-7.3.1-15.amzn2.x86_64                                  12/68 
2022-09-27T15:43:36.2928363Z   Installing : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       13/68 
2022-09-27T15:43:36.3163122Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 14/68 
2022-09-27T15:43:36.3503091Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         15/68 
2022-09-27T15:43:36.4079506Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        16/68 
2022-09-27T15:43:36.4656142Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   17/68 
2022-09-27T15:43:36.5748537Z   Installing : elfutils-0.176-2.amzn2.x86_64                              18/68 
2022-09-27T15:43:36.6022441Z   Installing : libsanitizer-7.3.1-15.amzn2.x86_64                         19/68 
2022-09-27T15:43:36.6269114Z   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 20/68 
2022-09-27T15:43:36.6576438Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          21/68 
2022-09-27T15:43:36.6891531Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        22/68 
2022-09-27T15:43:36.7746193Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                23/68 
2022-09-27T15:43:36.9378493Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         24/68 
2022-09-27T15:43:37.2297144Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           25/68 
2022-09-27T15:43:37.3983945Z   Installing : kernel-headers-4.14.291-218.527.amzn2.x86_64               26/68 
2022-09-27T15:43:37.5321949Z   Installing : glibc-headers-2.26-60.amzn2.x86_64                         27/68 
2022-09-27T15:43:37.5675925Z   Installing : glibc-devel-2.26-60.amzn2.x86_64                           28/68 
2022-09-27T15:43:37.9609968Z   Installing : libitm-7.3.1-15.amzn2.x86_64                               29/68 
2022-09-27T15:43:37.9890192Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              30/68 
2022-09-27T15:43:38.0146532Z   Installing : libmpx-7.3.1-15.amzn2.x86_64                               31/68 
2022-09-27T15:43:38.0433657Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      32/68 
2022-09-27T15:43:38.0661986Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               33/68 
2022-09-27T15:43:38.0881723Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  34/68 
2022-09-27T15:43:38.1802486Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  35/68 
2022-09-27T15:43:38.2253830Z   Installing : autoconf-2.69-11.amzn2.noarch                              36/68 
2022-09-27T15:43:38.3016673Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               37/68 
2022-09-27T15:43:38.3949898Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      38/68 
2022-09-27T15:43:38.5053284Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      39/68 
2022-09-27T15:43:38.5414204Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           40/68 
2022-09-27T15:43:38.5745655Z   Installing : libatomic-7.3.1-15.amzn2.x86_64                            41/68 
2022-09-27T15:43:40.6540429Z   Installing : libcilkrts-7.3.1-15.amzn2.x86_64                           42/68 
2022-09-27T15:43:44.5884062Z   Installing : gcc-7.3.1-15.amzn2.x86_64                                  43/68 
2022-09-27T15:43:55.7099337Z   Installing : kernel-devel-4.14.291-218.527.amzn2.x86_64                 44/68 
2022-09-27T15:43:56.3225672Z   Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     45/68 
2022-09-27T15:43:56.3748164Z   Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64                    46/68 
2022-09-27T15:43:56.4288043Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           47/68 
2022-09-27T15:43:56.5613405Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             48/68 
2022-09-27T15:43:56.7372326Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 49/68 
2022-09-27T15:43:56.8375993Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-09-27T15:43:58.0530783Z   Installing : systemtap-4.5-1.amzn2.0.1.x86_64                           51/68 
2022-09-27T15:43:59.6710550Z   Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64                         52/68 
2022-09-27T15:43:59.7902641Z   Installing : gcc-c++-7.3.1-15.amzn2.x86_64                              53/68 
2022-09-27T15:43:59.8267692Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        54/68 
2022-09-27T15:43:59.8646858Z   Installing : intltool-0.50.2-7.amzn2.noarch                             55/68 
2022-09-27T15:43:59.9197117Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       56/68 
2022-09-27T15:43:59.9772577Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            57/68 
2022-09-27T15:44:00.0805939Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             58/68 
2022-09-27T15:44:00.1433822Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             59/68 
2022-09-27T15:44:00.1990934Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               60/68 
2022-09-27T15:44:00.2423835Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              61/68 
2022-09-27T15:44:00.2874956Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          62/68 
2022-09-27T15:44:00.9855281Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        63/68 
2022-09-27T15:44:01.0268245Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             64/68 
2022-09-27T15:44:01.0509058Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           65/68 
2022-09-27T15:44:01.4022741Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      66/68 
2022-09-27T15:44:01.4408072Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            67/68 
2022-09-27T15:44:01.5041486Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        68/68 
2022-09-27T15:44:01.5153989Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                  1/68 
2022-09-27T15:44:01.5264759Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/68 
2022-09-27T15:44:01.5359388Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/68 
2022-09-27T15:44:01.5458706Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             4/68 
2022-09-27T15:44:01.5548538Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              5/68 
2022-09-27T15:44:01.5638318Z   Verifying  : glibc-headers-2.26-60.amzn2.x86_64                          6/68 
2022-09-27T15:44:01.5737487Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                            7/68 
2022-09-27T15:44:01.5825609Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         8/68 
2022-09-27T15:44:01.5916946Z   Verifying  : gcc-gfortran-7.3.1-15.amzn2.x86_64                          9/68 
2022-09-27T15:44:01.6029775Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            10/68 
2022-09-27T15:44:01.6116576Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      11/68 
2022-09-27T15:44:01.6222404Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                            12/68 
2022-09-27T15:44:01.6325476Z   Verifying  : libcilkrts-7.3.1-15.amzn2.x86_64                           13/68 
2022-09-27T15:44:01.6427140Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.1.noarch                  14/68 
2022-09-27T15:44:01.6511123Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        15/68 
2022-09-27T15:44:01.6590425Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            16/68 
2022-09-27T15:44:01.6684159Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           17/68 
2022-09-27T15:44:01.6788477Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             18/68 
2022-09-27T15:44:01.6879383Z   Verifying  : gcc-c++-7.3.1-15.amzn2.x86_64                              19/68 
2022-09-27T15:44:01.6987534Z   Verifying  : libatomic-7.3.1-15.amzn2.x86_64                            20/68 
2022-09-27T15:44:01.7081764Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               21/68 
2022-09-27T15:44:01.7190276Z   Verifying  : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     22/68 
2022-09-27T15:44:01.7277206Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      23/68 
2022-09-27T15:44:01.7378259Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              24/68 
2022-09-27T15:44:01.7468536Z   Verifying  : libquadmath-7.3.1-15.amzn2.x86_64                          25/68 
2022-09-27T15:44:01.7566039Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             26/68 
2022-09-27T15:44:01.7661164Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          27/68 
2022-09-27T15:44:01.7749499Z   Verifying  : glibc-devel-2.26-60.amzn2.x86_64                           28/68 
2022-09-27T15:44:01.7899251Z   Verifying  : kernel-devel-4.14.291-218.527.amzn2.x86_64                 29/68 
2022-09-27T15:44:01.7981916Z   Verifying  : cpp-7.3.1-15.amzn2.x86_64                                  30/68 
2022-09-27T15:44:01.8071682Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       31/68 
2022-09-27T15:44:01.8154438Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               32/68 
2022-09-27T15:44:01.8241133Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  33/68 
2022-09-27T15:44:01.8337582Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      34/68 
2022-09-27T15:44:01.8429028Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           35/68 
2022-09-27T15:44:01.8515781Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 36/68 
2022-09-27T15:44:01.8602598Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           37/68 
2022-09-27T15:44:01.8687295Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      38/68 
2022-09-27T15:44:01.8776480Z   Verifying  : libmpx-7.3.1-15.amzn2.x86_64                               39/68 
2022-09-27T15:44:01.8872154Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          40/68 
2022-09-27T15:44:01.8964085Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             41/68 
2022-09-27T15:44:01.9066991Z   Verifying  : libgfortran-7.3.1-15.amzn2.x86_64                          42/68 
2022-09-27T15:44:01.9181305Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              43/68 
2022-09-27T15:44:01.9262943Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        44/68 
2022-09-27T15:44:01.9358624Z   Verifying  : libitm-7.3.1-15.amzn2.x86_64                               45/68 
2022-09-27T15:44:01.9462182Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        46/68 
2022-09-27T15:44:01.9559057Z   Verifying  : gcc-7.3.1-15.amzn2.x86_64                                  47/68 
2022-09-27T15:44:01.9662924Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          48/68 
2022-09-27T15:44:01.9761271Z   Verifying  : kernel-headers-4.14.291-218.527.amzn2.x86_64               49/68 
2022-09-27T15:44:01.9863738Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-09-27T15:44:01.9956968Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   51/68 
2022-09-27T15:44:02.0044820Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              52/68 
2022-09-27T15:44:02.0125016Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        53/68 
2022-09-27T15:44:02.0217149Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              54/68 
2022-09-27T15:44:02.0309614Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         55/68 
2022-09-27T15:44:02.0398348Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             56/68 
2022-09-27T15:44:02.0502115Z   Verifying  : systemtap-4.5-1.amzn2.0.1.x86_64                           57/68 
2022-09-27T15:44:02.0586437Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                58/68 
2022-09-27T15:44:02.0686745Z   Verifying  : systemtap-client-4.5-1.amzn2.0.1.x86_64                    59/68 
2022-09-27T15:44:02.0778432Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          60/68 
2022-09-27T15:44:02.0871514Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               61/68 
2022-09-27T15:44:02.0978667Z   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 62/68 
2022-09-27T15:44:02.1069486Z   Verifying  : libsanitizer-7.3.1-15.amzn2.x86_64                         63/68 
2022-09-27T15:44:02.1161416Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              64/68 
2022-09-27T15:44:02.1259154Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              65/68 
2022-09-27T15:44:02.1347920Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   66/68 
2022-09-27T15:44:02.1435184Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         67/68 
2022-09-27T15:44:02.2175090Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.1.x86_64                       68/68 
2022-09-27T15:44:02.2175575Z 
2022-09-27T15:44:02.2175665Z Installed:
2022-09-27T15:44:02.2176061Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-09-27T15:44:02.2176501Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-09-27T15:44:02.2177792Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-09-27T15:44:02.2178317Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-09-27T15:44:02.2178763Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-09-27T15:44:02.2179173Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-09-27T15:44:02.2187413Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-09-27T15:44:02.2187886Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-09-27T15:44:02.2188308Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-09-27T15:44:02.2188868Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-09-27T15:44:02.2189677Z   gcc.x86_64 0:7.3.1-15.amzn2                                                   
2022-09-27T15:44:02.2190092Z   gcc-c++.x86_64 0:7.3.1-15.amzn2                                               
2022-09-27T15:44:02.2190525Z   gcc-gfortran.x86_64 0:7.3.1-15.amzn2                                          
2022-09-27T15:44:02.2190953Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-09-27T15:44:02.2191363Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-09-27T15:44:02.2191786Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-09-27T15:44:02.2192202Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-09-27T15:44:02.2192632Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-09-27T15:44:02.2193035Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-09-27T15:44:02.2193541Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-09-27T15:44:02.2194240Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-09-27T15:44:02.2194650Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-09-27T15:44:02.2195066Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-09-27T15:44:02.2195504Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14                                
2022-09-27T15:44:02.2195951Z   systemtap.x86_64 0:4.5-1.amzn2.0.1                                            
2022-09-27T15:44:02.2196141Z 
2022-09-27T15:44:02.2196264Z Dependency Installed:
2022-09-27T15:44:02.2196657Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-09-27T15:44:02.2197075Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-09-27T15:44:02.2197493Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-09-27T15:44:02.2197925Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-09-27T15:44:02.2198338Z   cpp.x86_64 0:7.3.1-15.amzn2                                                   
2022-09-27T15:44:02.2198750Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-09-27T15:44:02.2199152Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-09-27T15:44:02.2199595Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-09-27T15:44:02.2200059Z   emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                    
2022-09-27T15:44:02.2200776Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-09-27T15:44:02.2201224Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-09-27T15:44:02.2201870Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-09-27T15:44:02.2202297Z   glibc-devel.x86_64 0:2.26-60.amzn2                                            
2022-09-27T15:44:02.2202710Z   glibc-headers.x86_64 0:2.26-60.amzn2                                          
2022-09-27T15:44:02.2203131Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-09-27T15:44:02.2203671Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.1                                   
2022-09-27T15:44:02.2204128Z   kernel-devel.x86_64 0:4.14.291-218.527.amzn2                                  
2022-09-27T15:44:02.2204566Z   kernel-headers.x86_64 0:4.14.291-218.527.amzn2                                
2022-09-27T15:44:02.2204998Z   libatomic.x86_64 0:7.3.1-15.amzn2                                             
2022-09-27T15:44:02.2205416Z   libcilkrts.x86_64 0:7.3.1-15.amzn2                                            
2022-09-27T15:44:02.2205826Z   libgfortran.x86_64 0:7.3.1-15.amzn2                                           
2022-09-27T15:44:02.2206240Z   libitm.x86_64 0:7.3.1-15.amzn2                                                
2022-09-27T15:44:02.2206681Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-09-27T15:44:02.2207102Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-09-27T15:44:02.2207503Z   libmpx.x86_64 0:7.3.1-15.amzn2                                                
2022-09-27T15:44:02.2207915Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-09-27T15:44:02.2208338Z   libquadmath.x86_64 0:7.3.1-15.amzn2                                           
2022-09-27T15:44:02.2208739Z   libsanitizer.x86_64 0:7.3.1-15.amzn2                                          
2022-09-27T15:44:02.2209149Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-09-27T15:44:02.2209550Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-09-27T15:44:02.2209961Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-09-27T15:44:02.2210351Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-09-27T15:44:02.2210758Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-09-27T15:44:02.2211199Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-09-27T15:44:02.2211641Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-09-27T15:44:02.2212103Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-09-27T15:44:02.2212566Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-09-27T15:44:02.2213011Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-09-27T15:44:02.2213465Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-09-27T15:44:02.2213911Z   systemtap-client.x86_64 0:4.5-1.amzn2.0.1                                     
2022-09-27T15:44:02.2214352Z   systemtap-devel.x86_64 0:4.5-1.amzn2.0.1                                      
2022-09-27T15:44:02.2214762Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-09-27T15:44:02.2215181Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.1                                        
2022-09-27T15:44:02.2215386Z 
2022-09-27T15:44:02.2215491Z Complete!
2022-09-27T15:44:02.2571238Z ++ uname -r
2022-09-27T15:44:02.2577194Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-09-27T15:44:02.7663991Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:44:02.7774128Z Existing lock /var/run/yum.pid: another copy is running as pid 33176.
2022-09-27T15:44:02.7775374Z Another app is currently holding the yum lock; waiting for it to exit...
2022-09-27T15:44:02.7783359Z   The other application is: yum
2022-09-27T15:44:02.7784346Z     Memory :  87 M RSS (304 MB VSZ)
2022-09-27T15:44:02.7785130Z     Started: Tue Sep 27 15:44:01 2022 - 00:01 ago
2022-09-27T15:44:02.7785699Z     State  : Running, pid: 33176
2022-09-27T15:44:04.7810495Z Another app is currently holding the yum lock; waiting for it to exit...
2022-09-27T15:44:04.7816526Z   The other application is: yum
2022-09-27T15:44:04.7817104Z     Memory : 164 M RSS (382 MB VSZ)
2022-09-27T15:44:04.7818339Z     Started: Tue Sep 27 15:44:01 2022 - 00:03 ago
2022-09-27T15:44:04.7818937Z     State  : Running, pid: 33176
2022-09-27T15:44:07.0503515Z Resolving Dependencies
2022-09-27T15:44:07.0509547Z --> Running transaction check
2022-09-27T15:44:07.0510032Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-09-27T15:44:07.3360804Z --> Finished Dependency Resolution
2022-09-27T15:44:07.4180597Z 
2022-09-27T15:44:07.4181207Z Dependencies Resolved
2022-09-27T15:44:07.4186672Z 
2022-09-27T15:44:07.4186902Z ================================================================================
2022-09-27T15:44:07.4187305Z  Package          Arch       Version                       Repository      Size
2022-09-27T15:44:07.4187635Z ================================================================================
2022-09-27T15:44:07.4187920Z Installing:
2022-09-27T15:44:07.4188438Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-09-27T15:44:07.4188674Z 
2022-09-27T15:44:07.4188792Z Transaction Summary
2022-09-27T15:44:07.4189066Z ================================================================================
2022-09-27T15:44:07.4189342Z Install  1 Package
2022-09-27T15:44:07.4189501Z 
2022-09-27T15:44:07.4189631Z Total download size: 13 M
2022-09-27T15:44:07.4189970Z Installed size: 60 M
2022-09-27T15:44:07.4190791Z Downloading packages:
2022-09-27T15:44:07.4199307Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-09-27T15:44:07.7644669Z Running transaction check
2022-09-27T15:44:07.7831835Z Running transaction test
2022-09-27T15:44:08.1873518Z Transaction test succeeded
2022-09-27T15:44:08.1877363Z Running transaction
2022-09-27T15:44:23.5179647Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-09-27T15:44:23.5967676Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-09-27T15:44:23.5967985Z 
2022-09-27T15:44:23.5968098Z Installed:
2022-09-27T15:44:23.5968502Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-09-27T15:44:23.5968729Z 
2022-09-27T15:44:23.5968841Z Complete!
2022-09-27T15:44:23.6293701Z + sudo modprobe backlight
2022-09-27T15:44:23.6496942Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.57.run
2022-09-27T15:44:27.3132190Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-09-27T15:44:28.6814445Z Verifying archive integrity... OK
2022-09-27T15:44:55.3941325Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.57................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-09-27T15:44:55.5359719Z 
2022-09-27T15:44:55.5360447Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-09-27T15:44:55.5361063Z 
2022-09-27T15:45:10.9737872Z 
2022-09-27T15:45:10.9739117Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-09-27T15:45:10.9739912Z 
2022-09-27T15:45:20.0593001Z + sudo rm -fv /tmp/nvidia_driver
2022-09-27T15:45:20.1522939Z removed ‘/tmp/nvidia_driver’
2022-09-27T15:45:20.1536854Z + nvidia-smi
2022-09-27T15:45:24.7351369Z Tue Sep 27 15:45:24 2022       
2022-09-27T15:45:24.7351994Z +-----------------------------------------------------------------------------+
2022-09-27T15:45:24.7352566Z | NVIDIA-SMI 515.57       Driver Version: 515.57       CUDA Version: 11.7     |
2022-09-27T15:45:24.7353036Z |-------------------------------+----------------------+----------------------+
2022-09-27T15:45:24.7353558Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-09-27T15:45:24.7354078Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-09-27T15:45:24.7354464Z |                               |                      |               MIG M. |
2022-09-27T15:45:24.7354760Z |===============================+======================+======================|
2022-09-27T15:45:24.7398242Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |           8589873528 |
2022-09-27T15:45:24.7398635Z | N/A   25C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-09-27T15:45:24.7398968Z |                               |                      |                  N/A |
2022-09-27T15:45:24.7399477Z +-------------------------------+----------------------+----------------------+
2022-09-27T15:45:24.7444611Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
2022-09-27T15:45:24.7444986Z | N/A   28C    P0    37W / 150W |      0MiB /  7680MiB |     99%      Default |
2022-09-27T15:45:24.7445307Z |                               |                      |                  N/A |
2022-09-27T15:45:24.7445785Z +-------------------------------+----------------------+----------------------+
2022-09-27T15:45:24.7446148Z                                                                                
2022-09-27T15:45:24.7446584Z +-----------------------------------------------------------------------------+
2022-09-27T15:45:24.7446963Z | Processes:                                                                  |
2022-09-27T15:45:24.7447322Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-09-27T15:45:24.7447649Z |        ID   ID                                                   Usage      |
2022-09-27T15:45:24.7447957Z |=============================================================================|
2022-09-27T15:45:24.7450083Z |  No running processes found                                                 |
2022-09-27T15:45:24.7450605Z +-----------------------------------------------------------------------------+
2022-09-27T15:45:25.2774895Z == Installing nvidia container toolkit for amzn2 ==
2022-09-27T15:45:25.2778886Z + sudo yum install -y yum-utils
2022-09-27T15:45:25.8242039Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:45:26.0893684Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-09-27T15:45:26.0894069Z Nothing to do
2022-09-27T15:45:26.1097614Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-09-27T15:45:26.6688678Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:45:26.6974010Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-09-27T15:45:26.6974948Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-09-27T15:45:26.6975483Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-09-27T15:45:26.7120288Z + sudo yum install -y nvidia-docker2
2022-09-27T15:45:27.2581010Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-09-27T15:45:27.3019815Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-09-27T15:45:27.3203372Z Importing GPG key 0xF796ECB0:
2022-09-27T15:45:27.3203782Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-09-27T15:45:27.3204171Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-09-27T15:45:27.3204671Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-09-27T15:45:27.7121128Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-09-27T15:45:27.7206810Z Importing GPG key 0xF796ECB0:
2022-09-27T15:45:27.7207191Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-09-27T15:45:27.7207592Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-09-27T15:45:27.7208099Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-09-27T15:45:27.9463967Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-09-27T15:45:27.9549472Z Importing GPG key 0xF796ECB0:
2022-09-27T15:45:27.9549868Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-09-27T15:45:27.9550256Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-09-27T15:45:27.9550723Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-09-27T15:45:29.6325677Z Resolving Dependencies
2022-09-27T15:45:29.6332279Z --> Running transaction check
2022-09-27T15:45:29.6332753Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed
2022-09-27T15:45:29.6358374Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch
2022-09-27T15:45:29.6731357Z --> Running transaction check
2022-09-27T15:45:29.6731830Z ---> Package nvidia-container-toolkit.x86_64 0:1.11.0-1 will be installed
2022-09-27T15:45:29.6874762Z --> Processing Dependency: nvidia-container-toolkit-base = 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-09-27T15:45:29.6885683Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-09-27T15:45:29.7014761Z --> Processing Dependency: libnvidia-container-tools >= 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-09-27T15:45:29.7015267Z --> Running transaction check
2022-09-27T15:45:29.7016234Z ---> Package libnvidia-container-tools.x86_64 0:1.11.0-1 will be installed
2022-09-27T15:45:29.7026388Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.11.0-1 for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-09-27T15:45:29.7053804Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-09-27T15:45:29.7054527Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-09-27T15:45:29.7055307Z ---> Package nvidia-container-toolkit-base.x86_64 0:1.11.0-1 will be installed
2022-09-27T15:45:29.7057884Z --> Running transaction check
2022-09-27T15:45:29.7058338Z ---> Package libnvidia-container1.x86_64 0:1.11.0-1 will be installed
2022-09-27T15:45:29.9992316Z --> Finished Dependency Resolution
2022-09-27T15:45:30.0731867Z 
2022-09-27T15:45:30.0732194Z Dependencies Resolved
2022-09-27T15:45:30.0745339Z 
2022-09-27T15:45:30.0745633Z ================================================================================
2022-09-27T15:45:30.0745997Z  Package                        Arch    Version      Repository            Size
2022-09-27T15:45:30.0746342Z ================================================================================
2022-09-27T15:45:30.0746822Z Installing:
2022-09-27T15:45:30.0747298Z  nvidia-docker2                 noarch  2.11.0-1     libnvidia-container  8.7 k
2022-09-27T15:45:30.0747631Z Installing for dependencies:
2022-09-27T15:45:30.0748096Z  libnvidia-container-tools      x86_64  1.11.0-1     libnvidia-container   49 k
2022-09-27T15:45:30.0748719Z  libnvidia-container1           x86_64  1.11.0-1     libnvidia-container  1.0 M
2022-09-27T15:45:30.0749246Z  nvidia-container-toolkit       x86_64  1.11.0-1     libnvidia-container  780 k
2022-09-27T15:45:30.0749767Z  nvidia-container-toolkit-base  x86_64  1.11.0-1     libnvidia-container  2.5 M
2022-09-27T15:45:30.0750022Z 
2022-09-27T15:45:30.0751703Z Transaction Summary
2022-09-27T15:45:30.0752150Z ================================================================================
2022-09-27T15:45:30.0752501Z Install  1 Package (+4 Dependent packages)
2022-09-27T15:45:30.0752710Z 
2022-09-27T15:45:30.0752837Z Total download size: 4.3 M
2022-09-27T15:45:30.0753102Z Installed size: 12 M
2022-09-27T15:45:30.0753369Z Downloading packages:
2022-09-27T15:45:30.2059970Z --------------------------------------------------------------------------------
2022-09-27T15:45:30.2061036Z Total                                               33 MB/s | 4.3 MB  00:00     
2022-09-27T15:45:30.2106326Z Running transaction check
2022-09-27T15:45:30.2277016Z Running transaction test
2022-09-27T15:45:30.2439440Z Transaction test succeeded
2022-09-27T15:45:30.2443077Z Running transaction
2022-09-27T15:45:30.7310657Z   Installing : nvidia-container-toolkit-base-1.11.0-1.x86_64                1/5 
2022-09-27T15:45:30.7680017Z   Installing : libnvidia-container1-1.11.0-1.x86_64                         2/5 
2022-09-27T15:45:30.8764555Z   Installing : libnvidia-container-tools-1.11.0-1.x86_64                    3/5 
2022-09-27T15:45:30.9010505Z   Installing : nvidia-container-toolkit-1.11.0-1.x86_64                     4/5 
2022-09-27T15:45:30.9392135Z   Installing : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-09-27T15:45:30.9515099Z   Verifying  : libnvidia-container1-1.11.0-1.x86_64                         1/5 
2022-09-27T15:45:30.9608915Z   Verifying  : nvidia-container-toolkit-base-1.11.0-1.x86_64                2/5 
2022-09-27T15:45:30.9696720Z   Verifying  : nvidia-container-toolkit-1.11.0-1.x86_64                     3/5 
2022-09-27T15:45:30.9795347Z   Verifying  : libnvidia-container-tools-1.11.0-1.x86_64                    4/5 
2022-09-27T15:45:31.0533244Z   Verifying  : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-09-27T15:45:31.0533559Z 
2022-09-27T15:45:31.0533666Z Installed:
2022-09-27T15:45:31.0534334Z   nvidia-docker2.noarch 0:2.11.0-1                                              
2022-09-27T15:45:31.0534562Z 
2022-09-27T15:45:31.0534696Z Dependency Installed:
2022-09-27T15:45:31.0535126Z   libnvidia-container-tools.x86_64 0:1.11.0-1                                   
2022-09-27T15:45:31.0535605Z   libnvidia-container1.x86_64 0:1.11.0-1                                        
2022-09-27T15:45:31.0536098Z   nvidia-container-toolkit.x86_64 0:1.11.0-1                                    
2022-09-27T15:45:31.0537056Z   nvidia-container-toolkit-base.x86_64 0:1.11.0-1                               
2022-09-27T15:45:31.0537344Z 
2022-09-27T15:45:31.0537457Z Complete!
2022-09-27T15:45:31.1520609Z + sudo systemctl restart docker
2022-09-27T15:45:38.9242684Z + echo 'GPU_FLAG=--gpus all'
2022-09-27T15:45:39.4791072Z Command completed after 1 attempt(s).
2022-09-27T15:45:39.4791319Z 
2022-09-27T15:45:39.4848333Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-09-27T15:45:39.4848718Z python3 -m pip install psutil==5.9.1
2022-09-27T15:45:39.4849049Z python3 -m pip install pynvml==11.4.1
2022-09-27T15:45:39.4849404Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-09-27T15:45:39.4849769Z echo "::set-output name=monitor-script-pid::${!}"
2022-09-27T15:45:39.4863173Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:45:39.4863606Z env:
2022-09-27T15:45:39.4863849Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:39.4864107Z   GPU_FLAG: --gpus all
2022-09-27T15:45:39.4864356Z ##[endgroup]
2022-09-27T15:45:40.3814635Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T15:45:40.7434333Z Collecting psutil==5.9.1
2022-09-27T15:45:40.7635504Z   Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2022-09-27T15:45:40.8326166Z Installing collected packages: psutil
2022-09-27T15:45:40.9891025Z Successfully installed psutil-5.9.1
2022-09-27T15:45:41.4638253Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T15:45:41.5327113Z Collecting pynvml==11.4.1
2022-09-27T15:45:41.5492673Z   Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)
2022-09-27T15:45:41.5985847Z Installing collected packages: pynvml
2022-09-27T15:45:41.6515694Z Successfully installed pynvml-11.4.1
2022-09-27T15:45:41.7047754Z Prepare all required actions
2022-09-27T15:45:41.7048183Z Getting action download info
2022-09-27T15:45:41.9136183Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:ada9688bc02703b63dc0e606da280613803449a5)
2022-09-27T15:45:42.1320722Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-09-27T15:45:42.2542463Z ##[group]Run ./.github/actions/download-build-artifacts
2022-09-27T15:45:42.2542776Z with:
2022-09-27T15:45:42.2543048Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:45:42.2543330Z env:
2022-09-27T15:45:42.2543572Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:42.2543825Z   GPU_FLAG: --gpus all
2022-09-27T15:45:42.2544074Z ##[endgroup]
2022-09-27T15:45:42.2575811Z ##[group]Run seemethere/download-artifact-s3@v4
2022-09-27T15:45:42.2576106Z with:
2022-09-27T15:45:42.2576387Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:45:42.2576686Z   s3-bucket: gha-artifacts
2022-09-27T15:45:42.2576981Z   region: us-east-1
2022-09-27T15:45:42.2577214Z env:
2022-09-27T15:45:42.2577440Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:42.2577709Z   GPU_FLAG: --gpus all
2022-09-27T15:45:42.2577960Z ##[endgroup]
2022-09-27T15:45:42.7769823Z Found 1 objects with prefix pytorch/pytorch/3133193930/linux-bionic-cuda11.6-py3.10-gcc7/
2022-09-27T15:45:42.7770910Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-09-27T15:45:49.7938616Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-09-27T15:45:49.7938934Z 
2022-09-27T15:45:49.7939620Z Artifact download has finished successfully
2022-09-27T15:45:49.8085918Z ##[group]Run unzip -o artifacts.zip
2022-09-27T15:45:49.8086241Z unzip -o artifacts.zip
2022-09-27T15:45:49.8099696Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:45:49.8100004Z env:
2022-09-27T15:45:49.8100250Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:45:49.8100527Z   GPU_FLAG: --gpus all
2022-09-27T15:45:49.8100781Z ##[endgroup]
2022-09-27T15:45:49.8183418Z Archive:  artifacts.zip
2022-09-27T15:45:49.8185307Z    creating: dist/
2022-09-27T15:45:51.8498601Z   inflating: dist/torch-1.13.0a0+git52424e2-cp310-cp310-linux_x86_64.whl  
2022-09-27T15:45:51.8499043Z    creating: build/custom_test_artifacts/
2022-09-27T15:45:51.8499466Z    creating: build/custom_test_artifacts/custom-op-build/
2022-09-27T15:45:51.8499924Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-09-27T15:45:51.8506918Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:45:51.8507461Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-09-27T15:45:51.8508014Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:45:51.8508558Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:45:51.8509359Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:45:51.8511467Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:45:51.8512581Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:45:51.8513131Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:45:51.8513694Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:45:51.8516412Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:45:51.8517480Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:45:51.8519422Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:45:51.8520475Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:45:51.8521742Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:45:51.8522858Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:45:51.8523429Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-09-27T15:45:51.8524008Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-09-27T15:45:51.8578602Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-09-27T15:45:51.8579360Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-09-27T15:45:51.8580097Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-09-27T15:45:51.8580821Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-09-27T15:45:51.8581541Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-09-27T15:45:51.8582238Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-09-27T15:45:51.8582926Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-09-27T15:45:51.8583607Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-09-27T15:45:51.8584540Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-09-27T15:45:51.8626856Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-09-27T15:45:51.8668219Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-09-27T15:45:51.8669240Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-09-27T15:45:51.8669964Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-09-27T15:45:51.8670938Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-09-27T15:45:51.8671752Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-09-27T15:45:51.8672644Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-09-27T15:45:51.8673690Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-09-27T15:45:51.8675720Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-09-27T15:45:51.8748629Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-09-27T15:45:51.8821696Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-09-27T15:45:51.8822336Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-09-27T15:45:51.8822898Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-09-27T15:45:51.8823575Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-09-27T15:45:51.8824126Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:45:51.8824828Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-09-27T15:45:51.8825447Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-09-27T15:45:51.8826038Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-09-27T15:45:51.8826638Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-09-27T15:45:51.8827212Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-09-27T15:45:51.8827807Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-09-27T15:45:51.8829015Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-09-27T15:45:51.8829751Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-09-27T15:45:51.8830463Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-09-27T15:45:51.8831052Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-09-27T15:45:51.8852123Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-09-27T15:45:51.8965599Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-09-27T15:45:51.8966175Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-09-27T15:45:51.8966774Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-09-27T15:45:51.8967405Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-09-27T15:45:51.8968023Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-09-27T15:45:51.8968617Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-09-27T15:45:51.8969223Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-09-27T15:45:51.8970284Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-09-27T15:45:51.8971016Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-09-27T15:45:51.8971644Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-09-27T15:45:51.8972258Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-09-27T15:45:51.8993200Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-09-27T15:45:51.9074021Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-09-27T15:45:51.9074860Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:45:51.9075474Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:45:51.9076045Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-09-27T15:45:51.9076771Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-09-27T15:45:51.9077855Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:45:51.9078398Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-09-27T15:45:51.9081549Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-09-27T15:45:51.9082240Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-09-27T15:45:51.9083217Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-09-27T15:45:51.9176094Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-09-27T15:45:51.9237238Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-09-27T15:45:51.9237737Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-09-27T15:45:51.9238288Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-09-27T15:45:51.9245133Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:45:51.9245763Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-09-27T15:45:51.9246428Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:45:51.9247059Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:45:51.9247691Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:45:51.9248905Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:45:51.9250370Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:45:51.9251005Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:45:51.9251584Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:45:51.9254122Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:45:51.9255295Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:45:51.9256942Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:45:51.9257826Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:45:51.9259422Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:45:51.9260492Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:45:51.9261148Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-09-27T15:45:51.9261794Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-09-27T15:45:51.9315620Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-09-27T15:45:51.9316423Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-09-27T15:45:51.9317217Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-09-27T15:45:51.9318049Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-09-27T15:45:51.9318957Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-09-27T15:45:51.9319813Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-09-27T15:45:51.9320844Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-09-27T15:45:51.9321622Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-09-27T15:45:51.9322334Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-09-27T15:45:51.9363998Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-09-27T15:45:51.9405369Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-09-27T15:45:51.9406393Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-09-27T15:45:51.9407863Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-09-27T15:45:51.9408580Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-09-27T15:45:51.9409671Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-09-27T15:45:51.9410446Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-09-27T15:45:51.9411267Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-09-27T15:45:51.9413119Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-09-27T15:45:51.9486301Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-09-27T15:45:51.9559067Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-09-27T15:45:51.9559815Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-09-27T15:45:51.9560719Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-09-27T15:45:51.9561338Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-09-27T15:45:51.9561908Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:45:51.9562562Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-09-27T15:45:51.9563225Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-09-27T15:45:51.9563966Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-09-27T15:45:51.9564593Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-09-27T15:45:51.9565252Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-09-27T15:45:51.9565913Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-09-27T15:45:51.9566866Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-09-27T15:45:51.9567500Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-09-27T15:45:51.9604968Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-09-27T15:45:51.9605962Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-09-27T15:45:51.9606848Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-09-27T15:45:51.9652701Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-09-27T15:45:51.9653328Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:45:51.9653925Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:45:51.9654476Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-09-27T15:45:51.9655275Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-09-27T15:45:51.9656215Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:45:51.9656729Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-09-27T15:45:51.9659715Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-09-27T15:45:51.9660385Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-09-27T15:45:51.9661003Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-09-27T15:45:51.9710603Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-09-27T15:45:51.9711089Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-09-27T15:45:51.9711580Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-09-27T15:45:51.9718122Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-09-27T15:45:51.9718699Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-09-27T15:45:51.9719256Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-09-27T15:45:51.9719859Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-09-27T15:45:51.9720746Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-09-27T15:45:51.9722777Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-09-27T15:45:51.9723895Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-09-27T15:45:51.9724496Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-09-27T15:45:51.9725088Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-09-27T15:45:51.9727598Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-09-27T15:45:51.9728687Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-09-27T15:45:51.9730513Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-09-27T15:45:51.9731162Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-09-27T15:45:51.9732646Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-09-27T15:45:51.9733717Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-09-27T15:45:51.9734336Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-09-27T15:45:51.9734929Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-09-27T15:45:51.9789355Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-09-27T15:45:51.9790124Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-09-27T15:45:51.9791018Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-09-27T15:45:51.9791788Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-09-27T15:45:51.9792532Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-09-27T15:45:51.9793255Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-09-27T15:45:51.9793959Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-09-27T15:45:51.9794684Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-09-27T15:45:51.9795501Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-09-27T15:45:51.9837441Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-09-27T15:45:51.9878759Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-09-27T15:45:51.9879699Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-09-27T15:45:51.9880894Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-09-27T15:45:51.9881610Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-09-27T15:45:51.9882301Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-09-27T15:45:51.9883148Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-09-27T15:45:51.9884134Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-09-27T15:45:51.9886104Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-09-27T15:45:51.9959229Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-09-27T15:45:52.0032256Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-09-27T15:45:52.0032930Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-09-27T15:45:52.0033517Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-09-27T15:45:52.0034623Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-09-27T15:45:52.0035377Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-09-27T15:45:52.0035962Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-09-27T15:45:52.0036593Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-09-27T15:45:52.0037262Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-09-27T15:45:52.0037917Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-09-27T15:45:52.0038551Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-09-27T15:45:52.0039171Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-09-27T15:45:52.0040590Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-09-27T15:45:52.0041266Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-09-27T15:45:52.0041904Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-09-27T15:45:52.0042519Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-09-27T15:45:52.0046821Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-09-27T15:45:52.0193464Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-09-27T15:45:52.0194100Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-09-27T15:45:52.0194871Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-09-27T15:45:52.0195585Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-09-27T15:45:52.0196242Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-09-27T15:45:52.0196879Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-09-27T15:45:52.0197529Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-09-27T15:45:52.0198156Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-09-27T15:45:52.0198811Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-09-27T15:45:52.0199463Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-09-27T15:45:52.0200115Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-09-27T15:45:52.0220886Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-09-27T15:45:52.0278135Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-09-27T15:45:52.0278822Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-09-27T15:45:52.0279496Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-09-27T15:45:52.0280081Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-09-27T15:45:52.0280914Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-09-27T15:45:52.0282191Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-09-27T15:45:52.0282761Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-09-27T15:45:52.0285793Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-09-27T15:45:52.0286564Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-09-27T15:45:52.0287444Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-09-27T15:45:52.0406583Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-09-27T15:45:52.0451599Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-09-27T15:45:52.0451943Z    creating: build/lib/
2022-09-27T15:45:52.0452789Z   inflating: build/lib/libclog.a     
2022-09-27T15:45:52.0518849Z   inflating: build/lib/libgtest.a    
2022-09-27T15:45:52.0529319Z   inflating: build/lib/libpthreadpool.a  
2022-09-27T15:45:52.0620666Z   inflating: build/lib/libbenchmark.a  
2022-09-27T15:45:52.0726897Z   inflating: build/lib/libprotobuf-lite.a  
2022-09-27T15:45:52.0735921Z   inflating: build/lib/libittnotify.a  
2022-09-27T15:45:52.0767865Z   inflating: build/lib/libtensorpipe_uv.a  
2022-09-27T15:45:52.1301358Z   inflating: build/lib/libprotobuf.a  
2022-09-27T15:45:52.1378193Z   inflating: build/lib/libasmjit.a   
2022-09-27T15:45:52.1511150Z   inflating: build/lib/libgloo.a     
2022-09-27T15:45:52.1530945Z   inflating: build/lib/libfmt.a      
2022-09-27T15:45:52.1532843Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-09-27T15:45:52.1533276Z   inflating: build/lib/libfoxi_loader.a  
2022-09-27T15:45:52.1606402Z   inflating: build/lib/libc10.so     
2022-09-27T15:45:52.1607504Z   inflating: build/lib/libtorch_global_deps.so  
2022-09-27T15:45:52.1617647Z   inflating: build/lib/libcpuinfo.a  
2022-09-27T15:45:52.1626505Z   inflating: build/lib/libcpuinfo_internals.a  
2022-09-27T15:45:52.1642730Z   inflating: build/lib/libqnnpack.a  
2022-09-27T15:45:52.1666388Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-09-27T15:45:52.2236026Z   inflating: build/lib/libprotoc.a   
2022-09-27T15:45:52.2238574Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-09-27T15:45:52.2260871Z   inflating: build/lib/libnnpack.a   
2022-09-27T15:45:52.2279749Z   inflating: build/lib/libgmock.a    
2022-09-27T15:45:52.2280354Z   inflating: build/lib/libgtest_main.a  
2022-09-27T15:45:52.2281602Z   inflating: build/lib/libbenchmark_main.a  
2022-09-27T15:45:53.0398967Z   inflating: build/lib/libdnnl.a     
2022-09-27T15:45:53.1054026Z   inflating: build/lib/libtensorpipe.a  
2022-09-27T15:45:53.1195387Z   inflating: build/lib/libXNNPACK.a  
2022-09-27T15:45:53.1240286Z   inflating: build/lib/libc10_cuda.so  
2022-09-27T15:45:53.1241546Z   inflating: build/lib/libgmock_main.a  
2022-09-27T15:45:53.2767850Z   inflating: build/lib/libfbgemm.a   
2022-09-27T15:45:53.3057782Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-09-27T15:45:53.4186675Z   inflating: build/lib/libdnnl_graph.a  
2022-09-27T15:45:53.4608733Z   inflating: build/lib/libkineto.a   
2022-09-27T15:45:53.4654316Z   inflating: build/lib/libcaffe2_protos.a  
2022-09-27T15:45:53.4702240Z   inflating: build/lib/libonnx_proto.a  
2022-09-27T15:45:53.5380686Z   inflating: build/lib/libonnx.a     
2022-09-27T15:45:53.5809948Z   inflating: build/lib/libgloo_cuda.a  
2022-09-27T15:45:55.8836464Z   inflating: build/lib/libtorch_cpu.so  
2022-09-27T15:45:56.2268239Z   inflating: build/lib/libtorch_cuda_cpp.so  
2022-09-27T15:45:57.9250780Z   inflating: build/lib/libtorch_cuda_cu.so  
2022-09-27T15:45:57.9251592Z   inflating: build/lib/libtorch_cuda.so  
2022-09-27T15:45:57.9253229Z   inflating: build/lib/libtorch.so   
2022-09-27T15:45:57.9256744Z   inflating: build/lib/libc10d_cuda_test.so  
2022-09-27T15:45:58.9138294Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-09-27T15:45:58.9162917Z   inflating: build/lib/libjitbackend_test.so  
2022-09-27T15:45:58.9222891Z   inflating: build/lib/libtorchbind_test.so  
2022-09-27T15:45:58.9253516Z   inflating: build/lib/libbackend_with_compiler.so  
2022-09-27T15:45:58.9258139Z   inflating: build/lib/libshm.so     
2022-09-27T15:45:59.1038890Z   inflating: build/lib/libtorch_python.so  
2022-09-27T15:45:59.1078660Z   inflating: build/lib/libnnapi_backend.so  
2022-09-27T15:45:59.1078981Z    creating: build/bin/
2022-09-27T15:45:59.1131704Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-09-27T15:45:59.1187154Z   inflating: build/bin/c10_DeviceGuard_test  
2022-09-27T15:45:59.1240588Z   inflating: build/bin/c10_Device_test  
2022-09-27T15:45:59.1303819Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-09-27T15:45:59.1355184Z   inflating: build/bin/c10_StreamGuard_test  
2022-09-27T15:45:59.1409062Z   inflating: build/bin/c10_SymInt_test  
2022-09-27T15:45:59.1468560Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-09-27T15:45:59.1528522Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-09-27T15:45:59.1589365Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-09-27T15:45:59.1640560Z   inflating: build/bin/c10_Array_test  
2022-09-27T15:45:59.1697411Z   inflating: build/bin/c10_Bitset_test  
2022-09-27T15:45:59.1773141Z   inflating: build/bin/c10_C++17_test  
2022-09-27T15:45:59.1824461Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-09-27T15:45:59.1876612Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-09-27T15:45:59.1929754Z   inflating: build/bin/c10_Half_test  
2022-09-27T15:45:59.1990295Z   inflating: build/bin/c10_LeftRight_test  
2022-09-27T15:45:59.2057857Z   inflating: build/bin/c10_Metaprogramming_test  
2022-09-27T15:45:59.2213651Z   inflating: build/bin/c10_SmallVectorTest  
2022-09-27T15:45:59.2267191Z   inflating: build/bin/c10_Synchronized_test  
2022-09-27T15:45:59.2328490Z   inflating: build/bin/c10_ThreadLocal_test  
2022-09-27T15:45:59.2385834Z   inflating: build/bin/c10_TypeIndex_test  
2022-09-27T15:45:59.2439794Z   inflating: build/bin/c10_TypeList_test  
2022-09-27T15:45:59.2491305Z   inflating: build/bin/c10_TypeTraits_test  
2022-09-27T15:45:59.2546084Z   inflating: build/bin/c10_accumulate_test  
2022-09-27T15:45:59.2605832Z   inflating: build/bin/c10_bfloat16_test  
2022-09-27T15:45:59.2663386Z   inflating: build/bin/c10_complex_math_test  
2022-09-27T15:45:59.2722784Z   inflating: build/bin/c10_complex_test  
2022-09-27T15:45:59.2840254Z   inflating: build/bin/c10_either_test  
2022-09-27T15:45:59.2896166Z   inflating: build/bin/c10_exception_test  
2022-09-27T15:45:59.2949577Z   inflating: build/bin/c10_flags_test  
2022-09-27T15:45:59.3132030Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-09-27T15:45:59.3185880Z   inflating: build/bin/c10_irange_test  
2022-09-27T15:45:59.3247745Z   inflating: build/bin/c10_logging_test  
2022-09-27T15:45:59.3327554Z   inflating: build/bin/c10_optional_test  
2022-09-27T15:45:59.3393758Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-09-27T15:45:59.3452260Z   inflating: build/bin/c10_registry_test  
2022-09-27T15:45:59.3516025Z   inflating: build/bin/c10_string_view_test  
2022-09-27T15:45:59.3576431Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-09-27T15:45:59.3631240Z   inflating: build/bin/c10_tempfile_test  
2022-09-27T15:45:59.3691969Z   inflating: build/bin/c10_typeid_test  
2022-09-27T15:45:59.4213482Z   inflating: build/bin/protoc-3.13.0.0  
2022-09-27T15:45:59.4735929Z   inflating: build/bin/protoc        
2022-09-27T15:45:59.4787547Z   inflating: build/bin/c10_cuda_CUDATest  
2022-09-27T15:45:59.5104780Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-09-27T15:45:59.5458850Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-09-27T15:45:59.5523456Z   inflating: build/bin/TCPStoreTest  
2022-09-27T15:45:59.5580733Z   inflating: build/bin/FileStoreTest  
2022-09-27T15:45:59.5638113Z   inflating: build/bin/HashStoreTest  
2022-09-27T15:45:59.5654102Z   inflating: build/bin/ProcessGroupMPITest  
2022-09-27T15:45:59.5657431Z   inflating: build/bin/example_allreduce  
2022-09-27T15:45:59.5713300Z   inflating: build/bin/Dimname_test  
2022-09-27T15:45:59.5791517Z   inflating: build/bin/Dict_test     
2022-09-27T15:45:59.5859777Z   inflating: build/bin/MaybeOwned_test  
2022-09-27T15:45:59.5920739Z   inflating: build/bin/NamedTensor_test  
2022-09-27T15:45:59.5984273Z   inflating: build/bin/apply_utils_test  
2022-09-27T15:45:59.6047343Z   inflating: build/bin/atest         
2022-09-27T15:45:59.6111728Z   inflating: build/bin/basic         
2022-09-27T15:45:59.6169263Z   inflating: build/bin/broadcast_test  
2022-09-27T15:45:59.6231662Z   inflating: build/bin/cpu_generator_test  
2022-09-27T15:45:59.6288156Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-09-27T15:45:59.6341168Z   inflating: build/bin/dispatch_key_set_test  
2022-09-27T15:45:59.6437070Z   inflating: build/bin/cpu_rng_test  
2022-09-27T15:45:59.6490117Z   inflating: build/bin/dlconvertor_test  
2022-09-27T15:45:59.6552150Z   inflating: build/bin/extension_backend_test  
2022-09-27T15:45:59.6612025Z   inflating: build/bin/half_test     
2022-09-27T15:45:59.6664476Z   inflating: build/bin/lazy_tensor_test  
2022-09-27T15:45:59.6765157Z   inflating: build/bin/ivalue_test   
2022-09-27T15:45:59.6821018Z   inflating: build/bin/memory_format_test  
2022-09-27T15:45:59.6876959Z   inflating: build/bin/math_kernel_test  
2022-09-27T15:45:59.6933012Z   inflating: build/bin/memory_overlapping_test  
2022-09-27T15:45:59.6987122Z   inflating: build/bin/operator_name_test  
2022-09-27T15:45:59.7047025Z   inflating: build/bin/native_test   
2022-09-27T15:45:59.7102612Z   inflating: build/bin/mobile_memory_cleanup  
2022-09-27T15:45:59.7155933Z   inflating: build/bin/operators_test  
2022-09-27T15:45:59.7212145Z   inflating: build/bin/packedtensoraccessor_test  
2022-09-27T15:45:59.7281416Z   inflating: build/bin/pow_test      
2022-09-27T15:45:59.7342789Z   inflating: build/bin/quantized_test  
2022-09-27T15:45:59.7396778Z   inflating: build/bin/reportMemoryUsage_test  
2022-09-27T15:45:59.7449661Z   inflating: build/bin/reduce_ops_test  
2022-09-27T15:45:59.7509731Z   inflating: build/bin/scalar_tensor_test  
2022-09-27T15:45:59.7571055Z   inflating: build/bin/scalar_test   
2022-09-27T15:45:59.7626005Z   inflating: build/bin/stride_properties_test  
2022-09-27T15:45:59.7709716Z   inflating: build/bin/tensor_iterator_test  
2022-09-27T15:45:59.7769517Z   inflating: build/bin/type_ptr_test  
2022-09-27T15:45:59.7772178Z   inflating: build/bin/thread_init_test  
2022-09-27T15:45:59.7831423Z   inflating: build/bin/test_parallel  
2022-09-27T15:45:59.7887238Z   inflating: build/bin/undefined_tensor_test  
2022-09-27T15:45:59.7951732Z   inflating: build/bin/type_test     
2022-09-27T15:45:59.8004549Z   inflating: build/bin/variant_test  
2022-09-27T15:45:59.8005928Z   inflating: build/bin/verify_api_visibility  
2022-09-27T15:45:59.8080121Z   inflating: build/bin/vmap_test     
2022-09-27T15:45:59.8134315Z   inflating: build/bin/weakref_test  
2022-09-27T15:45:59.8188522Z   inflating: build/bin/wrapdim_test  
2022-09-27T15:45:59.8252633Z   inflating: build/bin/IListRef_test  
2022-09-27T15:45:59.8304704Z   inflating: build/bin/xla_tensor_test  
2022-09-27T15:45:59.8423991Z   inflating: build/bin/List_test     
2022-09-27T15:45:59.8556343Z   inflating: build/bin/kernel_function_legacy_test  
2022-09-27T15:45:59.8625387Z   inflating: build/bin/KernelFunction_test  
2022-09-27T15:45:59.8729226Z   inflating: build/bin/kernel_function_test  
2022-09-27T15:45:59.8867745Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-09-27T15:45:59.8980194Z   inflating: build/bin/kernel_lambda_test  
2022-09-27T15:45:59.9044381Z   inflating: build/bin/kernel_stackbased_test  
2022-09-27T15:45:59.9099393Z   inflating: build/bin/CppSignature_test  
2022-09-27T15:45:59.9202763Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-09-27T15:45:59.9254020Z   inflating: build/bin/op_allowlist_test  
2022-09-27T15:45:59.9310324Z   inflating: build/bin/inline_container_test  
2022-09-27T15:45:59.9623747Z   inflating: build/bin/op_registration_test  
2022-09-27T15:45:59.9684366Z   inflating: build/bin/backend_fallback_test  
2022-09-27T15:45:59.9740060Z   inflating: build/bin/cuda_apply_test  
2022-09-27T15:45:59.9806050Z   inflating: build/bin/cuda_atomic_ops_test  
2022-09-27T15:45:59.9863127Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-09-27T15:45:59.9915815Z   inflating: build/bin/cuda_device_test  
2022-09-27T15:45:59.9989088Z   inflating: build/bin/cuda_complex_math_test  
2022-09-27T15:46:00.0052156Z   inflating: build/bin/cuda_complex_test  
2022-09-27T15:46:00.0116395Z   inflating: build/bin/cuda_cub_test  
2022-09-27T15:46:00.0170004Z   inflating: build/bin/cuda_dlconvertor_test  
2022-09-27T15:46:00.0224655Z   inflating: build/bin/cuda_integer_divider_test  
2022-09-27T15:46:00.0297101Z   inflating: build/bin/cuda_distributions_test  
2022-09-27T15:46:00.0359833Z   inflating: build/bin/cuda_generator_test  
2022-09-27T15:46:00.0414351Z   inflating: build/bin/cuda_half_test  
2022-09-27T15:46:00.0466260Z   inflating: build/bin/cuda_optional_test  
2022-09-27T15:46:00.0531882Z   inflating: build/bin/cuda_stream_test  
2022-09-27T15:46:00.0587808Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-09-27T15:46:00.0643314Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-09-27T15:46:00.0695169Z   inflating: build/bin/cuda_cudnn_test  
2022-09-27T15:46:00.0751751Z   inflating: build/bin/cuda_vectorized_test  
2022-09-27T15:46:00.0768961Z   inflating: build/bin/tutorial_tensorexpr  
2022-09-27T15:46:00.0838409Z   inflating: build/bin/ProcessGroupGlooTest  
2022-09-27T15:46:00.0900693Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-09-27T15:46:00.0967267Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-09-27T15:46:00.1030397Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-09-27T15:46:00.1087341Z   inflating: build/bin/ProcessGroupUCCTest  
2022-09-27T15:46:00.1144915Z   inflating: build/bin/test_dist_autograd  
2022-09-27T15:46:00.1219584Z   inflating: build/bin/test_cpp_rpc  
2022-09-27T15:46:00.1222368Z   inflating: build/bin/parallel_benchmark  
2022-09-27T15:46:00.1296273Z   inflating: build/bin/test_mobile_nnc  
2022-09-27T15:46:00.1307445Z   inflating: build/bin/aot_model_compiler_test  
2022-09-27T15:46:00.2228380Z   inflating: build/bin/test_tensorexpr  
2022-09-27T15:46:00.2608588Z   inflating: build/bin/test_lazy     
2022-09-27T15:46:00.2614233Z   inflating: build/bin/torch_shm_manager  
2022-09-27T15:46:00.3919680Z   inflating: build/bin/test_api      
2022-09-27T15:46:00.5058699Z   inflating: build/bin/test_jit      
2022-09-27T15:46:00.5060510Z   inflating: .pytorch-test-times.json  
2022-09-27T15:46:00.5092625Z ##[group]Run df -H
2022-09-27T15:46:00.5092886Z df -H
2022-09-27T15:46:00.5105872Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T15:46:00.5106171Z env:
2022-09-27T15:46:00.5106417Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:46:00.5106669Z   GPU_FLAG: --gpus all
2022-09-27T15:46:00.5106918Z ##[endgroup]
2022-09-27T15:46:00.5147042Z Filesystem      Size  Used Avail Use% Mounted on
2022-09-27T15:46:00.5147691Z devtmpfs        129G     0  129G   0% /dev
2022-09-27T15:46:00.5148232Z tmpfs           129G     0  129G   0% /dev/shm
2022-09-27T15:46:00.5148752Z tmpfs           129G  549k  129G   1% /run
2022-09-27T15:46:00.5149208Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-09-27T15:46:00.5149511Z /dev/xvda1      162G   30G  132G  19% /
2022-09-27T15:46:00.5173498Z ##[group]Run .github/scripts/parse_ref.py
2022-09-27T15:46:00.5173858Z .github/scripts/parse_ref.py
2022-09-27T15:46:00.5185951Z shell: /usr/bin/bash -e {0}
2022-09-27T15:46:00.5186212Z env:
2022-09-27T15:46:00.5186457Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:46:00.5186727Z   GPU_FLAG: --gpus all
2022-09-27T15:46:00.5186960Z ##[endgroup]
2022-09-27T15:46:00.5504526Z ##[group]Run set -x
2022-09-27T15:46:00.5504928Z set -x
2022-09-27T15:46:00.5505167Z 
2022-09-27T15:46:00.5505448Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-09-27T15:46:00.5505795Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-09-27T15:46:00.5506150Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-09-27T15:46:00.5506481Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-09-27T15:46:00.5506744Z else
2022-09-27T15:46:00.5507030Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-09-27T15:46:00.5507313Z fi
2022-09-27T15:46:00.5507539Z 
2022-09-27T15:46:00.5507849Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-09-27T15:46:00.5508179Z 
2022-09-27T15:46:00.5508482Z # sanitize the input commit message and PR body here:
2022-09-27T15:46:00.5508763Z #
2022-09-27T15:46:00.5509144Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-09-27T15:46:00.5509667Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-09-27T15:46:00.5510256Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-09-27T15:46:00.5510579Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-09-27T15:46:00.5510823Z 
2022-09-27T15:46:00.5511184Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-09-27T15:46:00.5511569Z # wreak havoc internally
2022-09-27T15:46:00.5511880Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-09-27T15:46:00.5512218Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-09-27T15:46:00.5512489Z 
2022-09-27T15:46:00.5512787Z # detached container should get cleaned up by teardown_ec2_linux
2022-09-27T15:46:00.5513197Z # TODO: Stop building test binaries as part of the build phase
2022-09-27T15:46:00.5513580Z # Used for GPU_FLAG since that doesn't play nice
2022-09-27T15:46:00.5513916Z # shellcheck disable=SC2086,SC2090
2022-09-27T15:46:00.5514210Z container_name=$(docker run \
2022-09-27T15:46:00.5514498Z   ${GPU_FLAG:-} \
2022-09-27T15:46:00.5514776Z   -e BUILD_ENVIRONMENT \
2022-09-27T15:46:00.5515037Z   -e PR_NUMBER \
2022-09-27T15:46:00.5515309Z   -e GITHUB_ACTIONS \
2022-09-27T15:46:00.5515577Z   -e BASE_SHA \
2022-09-27T15:46:00.5515814Z   -e BRANCH \
2022-09-27T15:46:00.5516065Z   -e SHA1 \
2022-09-27T15:46:00.5516332Z   -e AWS_DEFAULT_REGION \
2022-09-27T15:46:00.5516613Z   -e IN_WHEEL_TEST \
2022-09-27T15:46:00.5516863Z   -e SHARD_NUMBER \
2022-09-27T15:46:00.5517127Z   -e TEST_CONFIG \
2022-09-27T15:46:00.5517397Z   -e NUM_TEST_SHARDS \
2022-09-27T15:46:00.5517644Z   -e PR_BODY \
2022-09-27T15:46:00.5517916Z   -e COMMIT_MESSAGES \
2022-09-27T15:46:00.5518217Z   -e PYTORCH_RETRY_TEST_CASES \
2022-09-27T15:46:00.5518522Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-09-27T15:46:00.5518820Z   -e PR_LABELS \
2022-09-27T15:46:00.5519116Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-09-27T15:46:00.5519393Z   -e SCCACHE_BUCKET \
2022-09-27T15:46:00.5519681Z   -e SCCACHE_S3_KEY_PREFIX \
2022-09-27T15:46:00.5519957Z   -e XLA_CUDA \
2022-09-27T15:46:00.5520654Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-09-27T15:46:00.5521024Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-09-27T15:46:00.5521351Z   --ulimit stack=10485760:83886080 \
2022-09-27T15:46:00.5521678Z   --security-opt seccomp=unconfined \
2022-09-27T15:46:00.5521973Z   --cap-add=SYS_PTRACE \
2022-09-27T15:46:00.5522248Z   --ipc=host \
2022-09-27T15:46:00.5522641Z   --shm-size="${SHM_SIZE}" \
2022-09-27T15:46:00.5522902Z   --tty \
2022-09-27T15:46:00.5523152Z   --detach \
2022-09-27T15:46:00.5523429Z   --name="${container_name}" \
2022-09-27T15:46:00.5523699Z   --user jenkins \
2022-09-27T15:46:00.5524028Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-09-27T15:46:00.5524379Z   -w /var/lib/jenkins/workspace \
2022-09-27T15:46:00.5524650Z   "${DOCKER_IMAGE}"
2022-09-27T15:46:00.5524900Z )
2022-09-27T15:46:00.5525242Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}"
2022-09-27T15:46:00.5536975Z shell: /usr/bin/bash -e {0}
2022-09-27T15:46:00.5537212Z env:
2022-09-27T15:46:00.5537460Z   GIT_DEFAULT_BRANCH: master
2022-09-27T15:46:00.5537738Z   GPU_FLAG: --gpus all
2022-09-27T15:46:00.5538053Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:46:00.5538375Z   PR_NUMBER: 85462
2022-09-27T15:46:00.5538631Z   BRANCH: pull/85462
2022-09-27T15:46:00.5538906Z   SHA1: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:46:00.5539237Z   BASE_SHA: 76d60778eb01b4213735be1c6e126fe2da519b8e
2022-09-27T15:46:00.5539546Z   PYTORCH_RETRY_TEST_CASES: 1
2022-09-27T15:46:00.5539935Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-09-27T15:46:00.5540222Z   TEST_CONFIG: distributed
2022-09-27T15:46:00.5540485Z   SHARD_NUMBER: 1
2022-09-27T15:46:00.5540715Z   NUM_TEST_SHARDS: 3
2022-09-27T15:46:00.5542971Z   PR_BODY: Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.

I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. 

Here is the code to reproduce the bug:

```
import torch
import weakref
from transformers import DetrForObjectDetection

class X:
    def __init__(self, optimizer):
        self.optimizer = optimizer

        # Will cause cyclic reference.
        self.func = self.dummy

        # Will work as expected, memory cleared after instance count is zero.
        # self.func = weakref.WeakMethod(self.dummy)

    def dummy(self, x):
        return 1.

def test():
    model = DetrForObjectDetection.from_pretrained('facebook/detr-resnet-50')
    model.to('cuda')
    optimizer = torch.optim.Adam(model.parameters())
    x = X(optimizer)

test()
print(f'{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}')  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).
```
2022-09-27T15:46:00.5545353Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-09-27T15:46:00.5545679Z   SCCACHE_S3_KEY_PREFIX: pull
2022-09-27T15:46:00.5545953Z   SHM_SIZE: 2g
2022-09-27T15:46:00.5546453Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:46:00.5546932Z   XLA_CUDA: 
2022-09-27T15:46:00.5547274Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-09-27T15:46:00.5547632Z ##[endgroup]
2022-09-27T15:46:00.5575817Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-09-27T15:46:00.5576300Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]]
2022-09-27T15:46:00.5576632Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-09-27T15:46:00.5579745Z ++ git cherry -v origin/master
2022-09-27T15:46:00.5823157Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR
2022-09-27T15:46:00.5823786Z + acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR
2022-09-27T15:46:00.5824191Z + 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form
2022-09-27T15:46:00.5824594Z + 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference
2022-09-27T15:46:00.5824949Z + 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:46:00.5826350Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:46:00.5843024Z + PR_BODY='Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained('\''facebook/detr-resnet-50'\'')    model.to('\''cuda'\'')    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f'\''{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()}'\'')  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:46:00.5846329Z + export 'COMMIT_MESSAGES=+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:46:00.5847751Z + COMMIT_MESSAGES='+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting'
2022-09-27T15:46:00.5854548Z + export 'PR_BODY=Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:46:00.5859950Z + PR_BODY='Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```'
2022-09-27T15:46:00.5862323Z +++ nproc --ignore=2
2022-09-27T15:46:00.5946711Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3133193930 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T15:46:14.2165992Z + container_name=28fd6574525b362559bd9225008ae2c88d5908bbdc045faa284102984d334c89
2022-09-27T15:46:14.2166829Z + docker exec -t 28fd6574525b362559bd9225008ae2c88d5908bbdc045faa284102984d334c89 sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh'
2022-09-27T15:46:14.8038122Z Processing ./dist/torch-1.13.0a0+git52424e2-cp310-cp310-linux_x86_64.whl
2022-09-27T15:46:14.8954812Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.13.0a0+git52424e2) (4.3.0)
2022-09-27T15:46:15.8418318Z Installing collected packages: torch
2022-09-27T15:46:26.4634303Z Successfully installed torch-1.13.0a0+git52424e2
2022-09-27T15:46:26.6329461Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-09-27T15:46:26.6555561Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-09-27T15:46:26.6556199Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-09-27T15:46:26.6556816Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-09-27T15:46:26.6557293Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-09-27T15:46:26.6559358Z + BUILD_DIR=build
2022-09-27T15:46:26.6559679Z + BUILD_RENAMED_DIR=build_renamed
2022-09-27T15:46:26.6559976Z + BUILD_BIN_DIR=build/bin
2022-09-27T15:46:26.6560704Z + export VALGRIND=ON
2022-09-27T15:46:26.6561098Z + VALGRIND=ON
2022-09-27T15:46:26.6561558Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *clang9* ]]
2022-09-27T15:46:26.6562020Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]]
2022-09-27T15:46:26.6562357Z ++ realpath build/custom_test_artifacts
2022-09-27T15:46:26.6570202Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-09-27T15:46:26.6573639Z ++ dirname .jenkins/pytorch/test.sh
2022-09-27T15:46:26.6580694Z + source .jenkins/pytorch/common.sh
2022-09-27T15:46:26.6584642Z +++ dirname .jenkins/pytorch/common.sh
2022-09-27T15:46:26.6595316Z ++ source .jenkins/pytorch/common_utils.sh
2022-09-27T15:46:26.6597306Z +++ declare -f -t trap_add
2022-09-27T15:46:26.6602216Z ++ set -ex
2022-09-27T15:46:26.6602919Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-09-27T15:46:26.6603278Z ++ BUILD_TEST_LIBTORCH=0
2022-09-27T15:46:26.6604139Z ++ [[ distributed == *xla* ]]
2022-09-27T15:46:26.6604561Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-09-27T15:46:26.6605027Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *linux-bionic* ]]
2022-09-27T15:46:26.6605341Z ++ which conda
2022-09-27T15:46:26.6613947Z /opt/conda/bin/conda
2022-09-27T15:46:26.6614860Z ++ conda install -q -y cmake
2022-09-27T15:46:29.3209876Z Collecting package metadata (current_repodata.json): ...working... done
2022-09-27T15:46:30.0477299Z Solving environment: ...working... done
2022-09-27T15:46:30.1534515Z 
2022-09-27T15:46:30.1534951Z ## Package Plan ##
2022-09-27T15:46:30.1535266Z 
2022-09-27T15:46:30.1535522Z   environment location: /opt/conda
2022-09-27T15:46:30.1535719Z 
2022-09-27T15:46:30.1535828Z   added / updated specs:
2022-09-27T15:46:30.1536221Z     - cmake
2022-09-27T15:46:30.1542153Z 
2022-09-27T15:46:30.1542360Z 
2022-09-27T15:46:30.1542855Z The following packages will be downloaded:
2022-09-27T15:46:30.1543186Z 
2022-09-27T15:46:30.1543332Z     package                    |            build
2022-09-27T15:46:30.1543780Z     ---------------------------|-----------------
2022-09-27T15:46:30.1544478Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-09-27T15:46:30.1544922Z     certifi-2022.9.14          |  py310h06a4308_0         155 KB
2022-09-27T15:46:30.1545329Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-09-27T15:46:30.1545744Z     conda-22.9.0               |  py310h06a4308_0         894 KB
2022-09-27T15:46:30.1546200Z     expat-2.4.4                |       h295c915_0         169 KB
2022-09-27T15:46:30.1546751Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-09-27T15:46:30.1547143Z     libcurl-7.82.0             |       h0b77cf5_0         342 KB
2022-09-27T15:46:30.1547546Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-09-27T15:46:30.1547918Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-09-27T15:46:30.1548322Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-09-27T15:46:30.1548747Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-09-27T15:46:30.1549115Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-09-27T15:46:30.1549743Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-09-27T15:46:30.1550157Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-09-27T15:46:30.1550547Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-09-27T15:46:30.1550938Z     ------------------------------------------------------------
2022-09-27T15:46:30.1551279Z                                            Total:        12.9 MB
2022-09-27T15:46:30.1551460Z 
2022-09-27T15:46:30.1551623Z The following NEW packages will be INSTALLED:
2022-09-27T15:46:30.1551827Z 
2022-09-27T15:46:30.1552160Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0
2022-09-27T15:46:30.1552649Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0
2022-09-27T15:46:30.1553120Z   expat              pkgs/main/linux-64::expat-2.4.4-h295c915_0
2022-09-27T15:46:30.1553591Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0
2022-09-27T15:46:30.1554051Z   libcurl            pkgs/main/linux-64::libcurl-7.82.0-h0b77cf5_0
2022-09-27T15:46:30.1554558Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0
2022-09-27T15:46:30.1555039Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
2022-09-27T15:46:30.1555675Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0
2022-09-27T15:46:30.1556160Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0
2022-09-27T15:46:30.1556640Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
2022-09-27T15:46:30.1557123Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1
2022-09-27T15:46:30.1557569Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1
2022-09-27T15:46:30.1558034Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0
2022-09-27T15:46:30.1558242Z 
2022-09-27T15:46:30.1558389Z The following packages will be UPDATED:
2022-09-27T15:46:30.1558582Z 
2022-09-27T15:46:30.1558869Z   certifi                         2022.6.15-py310h06a4308_0 --> 2022.9.14-py310h06a4308_0
2022-09-27T15:46:30.1559321Z   conda                              4.14.0-py310h06a4308_0 --> 22.9.0-py310h06a4308_0
2022-09-27T15:46:30.1559534Z 
2022-09-27T15:46:30.1559558Z 
2022-09-27T15:46:31.3969134Z Preparing transaction: ...working... done
2022-09-27T15:46:31.9096295Z Verifying transaction: ...working... done
2022-09-27T15:46:33.2713850Z Executing transaction: ...working... done
2022-09-27T15:46:33.4432189Z Retrieving notices: ...working... done
2022-09-27T15:46:33.6450145Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-09-27T15:46:33.6450550Z + echo 'Environment variables'
2022-09-27T15:46:33.6450846Z Environment variables
2022-09-27T15:46:33.6451094Z + env
2022-09-27T15:46:33.6458953Z SHARD_NUMBER=1
2022-09-27T15:46:33.6459548Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2022-09-27T15:46:33.6459990Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2022-09-27T15:46:33.6460380Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-09-27T15:46:33.6460841Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2022-09-27T15:46:33.6461119Z UCC_HOME=/usr
2022-09-27T15:46:33.6461508Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T15:46:33.6461958Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2022-09-27T15:46:33.6462240Z INSTALLED_DB=yes
2022-09-27T15:46:33.6462500Z HOSTNAME=28fd6574525b
2022-09-27T15:46:33.6462764Z GITHUB_REF_NAME=85462/merge
2022-09-27T15:46:33.6463064Z GITHUB_API_URL=https://api.github.com
2022-09-27T15:46:33.6463369Z OPENSSL_DIR=/opt/openssl
2022-09-27T15:46:33.6463683Z UCC_COMMIT=12944da33f911daf505d9bbc51411233d0ed85e1
2022-09-27T15:46:33.6464262Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_02198a53-f9ad-4214-8ef0-55ed8b3da472
2022-09-27T15:46:33.6464680Z CUDA_PATH=/usr/local/cuda
2022-09-27T15:46:33.6465174Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-09-27T15:46:33.6465555Z GITHUB_RUN_ATTEMPT=2
2022-09-27T15:46:33.6466073Z TEST_CONFIG=distributed
2022-09-27T15:46:33.6466412Z NV_LIBNPP_VERSION=11.6.3.124-1
2022-09-27T15:46:33.6467083Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2022-09-27T15:46:33.6467575Z GITHUB_REPOSITORY_OWNER=pytorch
2022-09-27T15:46:33.6467985Z GITHUB_ACTIONS=true
2022-09-27T15:46:33.6468382Z NVIDIA_VISIBLE_DEVICES=all
2022-09-27T15:46:33.6468891Z NV_NVPROF_VERSION=11.6.124-1
2022-09-27T15:46:33.6469201Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2022-09-27T15:46:33.6469464Z CI=true
2022-09-27T15:46:33.6469718Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-09-27T15:46:33.6470102Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2022-09-27T15:46:33.6470413Z BRANCH=pull/85462
2022-09-27T15:46:33.6470739Z GITHUB_HEAD_REF=cycliclr-memory-fix
2022-09-27T15:46:33.6471069Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2022-09-27T15:46:33.6471384Z GITHUB_ACTOR=kongzii
2022-09-27T15:46:33.6471686Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-09-27T15:46:33.6471960Z GITHUB_ACTION_REF=
2022-09-27T15:46:33.6472240Z NCCL_VERSION=2.12.10-1
2022-09-27T15:46:33.6472495Z GITHUB_ACTION=__self
2022-09-27T15:46:33.6472722Z VALGRIND=ON
2022-09-27T15:46:33.6472976Z GITHUB_REF_PROTECTED=false
2022-09-27T15:46:33.6473589Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-09-27T15:46:33.6476105Z ***
2022-09-27T15:46:33.6476395Z INSTALLED_VISION=yes
2022-09-27T15:46:33.6476657Z NVARCH=x86_64
2022-09-27T15:46:33.6476991Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2022-09-27T15:46:33.6477291Z HOME=/var/lib/jenkins
2022-09-27T15:46:33.6477576Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-09-27T15:46:33.6477875Z GITHUB_ACTION_REPOSITORY=
2022-09-27T15:46:33.6478139Z GITHUB_REF_TYPE=branch
2022-09-27T15:46:33.6478465Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2022-09-27T15:46:33.6478765Z GITHUB_RETENTION_DAYS=90
2022-09-27T15:46:33.6479157Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-09-27T15:46:33.6479594Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2022-09-27T15:46:33.6480627Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_02198a53-f9ad-4214-8ef0-55ed8b3da472
2022-09-27T15:46:33.6481232Z DEBIAN_FRONTEND=noninteractive
2022-09-27T15:46:33.6481729Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-09-27T15:46:33.6482119Z GITHUB_REF=refs/pull/85462/merge
2022-09-27T15:46:33.6482416Z NV_CUDA_LIB_VERSION=11.6.2-1
2022-09-27T15:46:33.6482739Z GITHUB_SHA=1faa2af6dbb8dd899ab20874e9966185467c5883
2022-09-27T15:46:33.6483047Z INSTALLED_PROTOBUF=yes
2022-09-27T15:46:33.6483293Z GITHUB_RUN_ID=3133193930
2022-09-27T15:46:33.6483648Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2022-09-27T15:46:33.6483955Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-09-27T15:46:33.6484283Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-09-27T15:46:33.6484600Z NV_NVTX_VERSION=11.6.124-1
2022-09-27T15:46:33.6484903Z GITHUB_SERVER_URL=https://github.com
2022-09-27T15:46:33.6485163Z MAX_JOBS=30
2022-09-27T15:46:33.6485452Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2022-09-27T15:46:33.6485833Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2022-09-27T15:46:33.6486325Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-09-27T15:46:33.6486653Z UCX_HOME=/usr
2022-09-27T15:46:33.6486914Z PYTORCH_RETRY_TEST_CASES=1
2022-09-27T15:46:33.6487245Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-09-27T15:46:33.6487580Z BASE_SHA=76d60778eb01b4213735be1c6e126fe2da519b8e
2022-09-27T15:46:33.6487918Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2022-09-27T15:46:33.6491491Z PR_BODY=Hi, we noticed in our team that by using CyclicLR, there is a problem with memory clearance on GPU (probably it will be the case without the GPU as well, but that was our use case) After initializing CyclicLR, GPU memory is not cleared even after the model, optimizer and scheduler are out of scope (e.g. reference count is zero). This is because `__init__` method inside `CyclicLR` creates reference to its own methods and it will not get removed until `gc.collect()` is called manually. This is a problem if people want to test multiple models in one run of a script, after testing the first model, second one will fail on `CUDA out of memory error` because the first one is not cleared from the memory.I propose a simple fix by using `weakref`, similarly as in `_LRScheduler` base class, but if you have any comments I am happy to change it. Here is the code to reproduce the bug:```import torchimport weakreffrom transformers import DetrForObjectDetectionclass X:    def __init__(self, optimizer):        self.optimizer = optimizer        # Will cause cyclic reference.        self.func = self.dummy        # Will work as expected, memory cleared after instance count is zero.        # self.func = weakref.WeakMethod(self.dummy)    def dummy(self, x):        return 1.def test():    model = DetrForObjectDetection.from_pretrained(facebook/detr-resnet-50)    model.to(cuda)    optimizer = torch.optim.Adam(model.parameters())    x = X(optimizer)test()print(f{torch.cuda.memory_reserved()}, {torch.cuda.memory_allocated()})  # Should print (<some memory>, 0), but with cyclic reference, it will print (<some memory>, <some memory>).```
2022-09-27T15:46:33.6493816Z GITHUB_BASE_REF=master
2022-09-27T15:46:33.6494051Z TERM=xterm
2022-09-27T15:46:33.6494362Z XLA_CUDA=
2022-09-27T15:46:33.6494652Z NV_NVML_DEV_VERSION=11.6.55-1
2022-09-27T15:46:33.6494916Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-09-27T15:46:33.6495183Z CUDA_VERSION=11.6.2
2022-09-27T15:46:33.6495530Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2022-09-27T15:46:33.6495835Z OPENSSL_ROOT_DIR=/opt/openssl
2022-09-27T15:46:33.6496366Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_02198a53-f9ad-4214-8ef0-55ed8b3da472
2022-09-27T15:46:33.6496749Z GITHUB_JOB=test
2022-09-27T15:46:33.6497014Z SCCACHE_S3_KEY_PREFIX=pull
2022-09-27T15:46:33.6497701Z COMMIT_MESSAGES=+ 871567eae42e57e9926f1a38c0b8d221f672c928 Fix mem leak because of self reference in CyclicLR+ acca92f8843f34854e73ee280c4f87ea280d2914 Typing for new CyclicLR+ 788218632109bd2065d4961cae624c62f106deee Rename scale_fn_* to private form+ 1642fe39e72ecf43de500a53610a076827e792b9 Test CyclicLR cyclic reference+ 52424e2bf38e454d535881fed9628d3e20f4f944 Fix linting
2022-09-27T15:46:33.6498406Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-09-27T15:46:33.6498705Z NUM_TEST_SHARDS=3
2022-09-27T15:46:33.6498951Z PR_NUMBER=85462
2022-09-27T15:46:33.6499168Z SHLVL=1
2022-09-27T15:46:33.6499518Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2022-09-27T15:46:33.6499848Z GITHUB_REPOSITORY=pytorch/pytorch
2022-09-27T15:46:33.6500431Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471
2022-09-27T15:46:33.6501028Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2022-09-27T15:46:33.6501338Z SHA1=52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T15:46:33.6501642Z GITHUB_EVENT_NAME=pull_request
2022-09-27T15:46:33.6501945Z NV_CUDA_CUDART_VERSION=11.6.55-1
2022-09-27T15:46:33.6502295Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-09-27T15:46:33.6502587Z GITHUB_RUN_NUMBER=50832
2022-09-27T15:46:33.6502838Z GITHUB_WORKFLOW=pull
2022-09-27T15:46:33.6503258Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:46:33.6503709Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2022-09-27T15:46:33.6504132Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T15:46:33.6504488Z GITHUB_TRIGGERING_ACTOR=albanD
2022-09-27T15:46:33.6504751Z _=/usr/bin/env
2022-09-27T15:46:33.6505024Z + echo 'Testing pytorch'
2022-09-27T15:46:33.6505281Z Testing pytorch
2022-09-27T15:46:33.6505555Z + export LANG=C.UTF-8
2022-09-27T15:46:33.6505824Z + LANG=C.UTF-8
2022-09-27T15:46:33.6506064Z + PR_NUMBER=85462
2022-09-27T15:46:33.6506339Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-09-27T15:46:33.6506711Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-09-27T15:46:33.6507120Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-09-27T15:46:33.6507450Z + [[ distributed == \s\l\o\w ]]
2022-09-27T15:46:33.6507870Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]]
2022-09-27T15:46:33.6508294Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-09-27T15:46:33.6508641Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-09-27T15:46:33.6508962Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-09-27T15:46:33.6509357Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda11* ]]
2022-09-27T15:46:33.6509681Z + export BUILD_SPLIT_CUDA=ON
2022-09-27T15:46:33.6509950Z + BUILD_SPLIT_CUDA=ON
2022-09-27T15:46:33.6510204Z + [[ distributed == *crossref* ]]
2022-09-27T15:46:33.6510490Z + [[ distributed == *dynamo* ]]
2022-09-27T15:46:33.6510781Z + [[ -n 85462 ]]
2022-09-27T15:46:33.6511021Z + [[ -z '' ]]
2022-09-27T15:46:33.6511312Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-09-27T15:46:33.6511642Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-09-27T15:46:33.6512032Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-09-27T15:46:33.6512483Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]]
2022-09-27T15:46:33.6512916Z + pip_install --user ninja
2022-09-27T15:46:33.6513281Z + pip install --progress-bar off --user ninja
2022-09-27T15:46:34.2236531Z Collecting ninja
2022-09-27T15:46:34.2497355Z   Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-09-27T15:46:35.1051922Z Installing collected packages: ninja
2022-09-27T15:46:35.1154693Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-09-27T15:46:35.1155372Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T15:46:35.1209229Z Successfully installed ninja-1.10.2.3
2022-09-27T15:46:35.1928976Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:46:35.1929627Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-09-27T15:46:35.1930276Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]]
2022-09-27T15:46:35.1930622Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-09-27T15:46:35.1930953Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-09-27T15:46:35.1937540Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]]
2022-09-27T15:46:35.1950339Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-09-27T15:46:35.1950781Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]]
2022-09-27T15:46:35.1953990Z + cd test
2022-09-27T15:46:35.1954379Z + python -c 'import torch; print(torch.__config__.show())'
2022-09-27T15:46:36.8343661Z PyTorch built with:
2022-09-27T15:46:36.8344156Z   - GCC 7.5
2022-09-27T15:46:36.8344453Z   - C++ Version: 201402
2022-09-27T15:46:36.8345009Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-09-27T15:46:36.8345586Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-09-27T15:46:36.8345997Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-09-27T15:46:36.8346392Z   - LAPACK is enabled (usually provided by MKL)
2022-09-27T15:46:36.8346724Z   - NNPACK is enabled
2022-09-27T15:46:36.8347023Z   - CPU capability usage: AVX2
2022-09-27T15:46:36.8347334Z   - CUDA Runtime 11.6
2022-09-27T15:46:36.8347727Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-09-27T15:46:36.8348128Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2022-09-27T15:46:36.8348415Z   - Magma 2.6.1
2022-09-27T15:46:36.8351553Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-09-27T15:46:36.8353599Z 
2022-09-27T15:46:37.0570742Z + cd test
2022-09-27T15:46:37.0571338Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-09-27T15:46:38.6306247Z ATen/Parallel:
2022-09-27T15:46:38.6307439Z 	at::get_num_threads() : 16
2022-09-27T15:46:38.6307938Z 	at::get_num_interop_threads() : 16
2022-09-27T15:46:38.6308250Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-09-27T15:46:38.6308535Z 	omp_get_max_threads() : 16
2022-09-27T15:46:38.6309203Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-09-27T15:46:38.6309581Z 	mkl_get_max_threads() : 16
2022-09-27T15:46:38.6310038Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-09-27T15:46:38.6310418Z std::thread::hardware_concurrency() : 32
2022-09-27T15:46:38.6310695Z Environment variables:
2022-09-27T15:46:38.6310971Z 	OMP_NUM_THREADS : [not set]
2022-09-27T15:46:38.6311243Z 	MKL_NUM_THREADS : [not set]
2022-09-27T15:46:38.6311514Z ATen parallel backend: OpenMP
2022-09-27T15:46:38.6311698Z 
2022-09-27T15:46:38.8429047Z + [[ distributed == *deploy* ]]
2022-09-27T15:46:38.8429803Z + [[ distributed == *backward* ]]
2022-09-27T15:46:38.8430431Z + [[ distributed == *xla* ]]
2022-09-27T15:46:38.8430776Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-09-27T15:46:38.8431304Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-09-27T15:46:38.8431643Z + [[ distributed == distributed ]]
2022-09-27T15:46:38.8431922Z + install_torchdynamo
2022-09-27T15:46:38.8432151Z + local commit
2022-09-27T15:46:38.8432818Z ++ get_pinned_commit torchdynamo
2022-09-27T15:46:38.8433550Z ++ cat .github/ci_commit_pins/torchdynamo.txt
2022-09-27T15:46:38.8449142Z + commit=41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:38.8450330Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:38.8451061Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:39.3390420Z Collecting git+https://github.com/pytorch/torchdynamo.git@41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:39.3396574Z   Cloning https://github.com/pytorch/torchdynamo.git (to revision 41c44bc1d080d6cf063419a4166732b983b84eef) to /tmp/pip-req-build-0529eayz
2022-09-27T15:46:39.3417490Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-0529eayz
2022-09-27T15:46:40.1495776Z   Running command git rev-parse -q --verify 'sha^41c44bc1d080d6cf063419a4166732b983b84eef'
2022-09-27T15:46:40.1516380Z   Running command git fetch -q https://github.com/pytorch/torchdynamo.git 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:40.4870996Z   Running command git checkout -q 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:40.7876845Z   Resolved https://github.com/pytorch/torchdynamo.git to commit 41c44bc1d080d6cf063419a4166732b983b84eef
2022-09-27T15:46:43.2105107Z   Preparing metadata (setup.py) ... [?25l- done
2022-09-27T15:46:43.2173295Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+git52424e2)
2022-09-27T15:46:43.2177745Z Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.21.2)
2022-09-27T15:46:43.2611285Z Collecting tabulate
2022-09-27T15:46:43.2803806Z   Downloading tabulate-0.8.10-py3-none-any.whl (29 kB)
2022-09-27T15:46:43.2863797Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.10/site-packages/PyYAML-6.0-py3.10-linux-x86_64.egg (from torchdynamo==1.13.0.dev0) (6.0)
2022-09-27T15:46:43.2868582Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.11.1)
2022-09-27T15:46:43.2894512Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0)
2022-09-27T15:46:43.2927705Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1)
2022-09-27T15:46:43.3052296Z Building wheels for collected packages: torchdynamo
2022-09-27T15:46:47.8349335Z   Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - done
2022-09-27T15:46:47.8446045Z [?25h  Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp310-cp310-linux_x86_64.whl size=2600957 sha256=222389bf3fa27df50805d425b87fc47beb50a6d6739773f3065519c54b02ba20
2022-09-27T15:46:47.8448320Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/2e/47/4b/a72e6a8c4801cae81c62fd871ce3601d87ba0b7e2d5534e15c
2022-09-27T15:46:47.8473476Z Successfully built torchdynamo
2022-09-27T15:46:48.6780677Z Installing collected packages: tabulate, torchdynamo
2022-09-27T15:46:49.0335029Z Successfully installed tabulate-0.8.10 torchdynamo-1.13.0.dev0
2022-09-27T15:46:49.1221434Z + test_distributed
2022-09-27T15:46:49.1221956Z + echo 'Testing distributed python tests'
2022-09-27T15:46:49.1222281Z Testing distributed python tests
2022-09-27T15:46:49.1222710Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose
2022-09-27T15:46:51.3019650Z Ignoring disabled issues:  []
2022-09-27T15:46:51.3377337Z /var/lib/jenkins/workspace/test/run_test.py:960: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-09-27T15:46:51.3378094Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-09-27T15:46:51.3379452Z Found test time stats from artifacts
2022-09-27T15:46:51.3381749Z Selected tests:
2022-09-27T15:46:51.3382136Z  distributed/test_distributed_spawn
2022-09-27T15:46:51.3382442Z  distributed/_shard/test_sharder
2022-09-27T15:46:51.3382761Z  distributed/fsdp/test_shard_utils
2022-09-27T15:46:51.3383345Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-09-27T15:46:51.3383718Z  distributed/pipeline/sync/skip/test_stash_pop
2022-09-27T15:46:51.3384064Z  distributed/pipeline/sync/test_balance
2022-09-27T15:46:51.3384603Z  distributed/pipeline/sync/test_copy
2022-09-27T15:46:51.3384932Z  distributed/pipeline/sync/test_inplace
2022-09-27T15:46:51.3385257Z  distributed/pipeline/sync/test_pipe
2022-09-27T15:46:51.3385672Z  distributed/pipeline/sync/test_transparency
2022-09-27T15:46:51.3386178Z  distributed/rpc/test_tensorpipe_agent
2022-09-27T15:46:51.3513591Z Prioritized test from test file changes.
2022-09-27T15:46:51.3514043Z reordering tests for PR:
2022-09-27T15:46:51.3514428Z prioritized: []
2022-09-27T15:46:51.3516211Z the rest: ['distributed/test_distributed_spawn', 'distributed/_shard/test_sharder', 'distributed/fsdp/test_shard_utils', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_transparency', 'distributed/rpc/test_tensorpipe_agent']
2022-09-27T15:46:51.3517285Z 
2022-09-27T15:46:51.3517859Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-09-27T15:46:51.3758354Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-09-27T15:46:51.4659167Z Running distributed/test_distributed_spawn ... [2022-09-27 15:46:51.465576]
2022-09-27T15:46:51.4667663Z /usr/bin/mpiexec
2022-09-27T15:46:51.4668638Z MPI not available -- MPI backend tests will be skipped
2022-09-27T15:46:51.4868432Z Running distributed tests for the test backend with env init_method
2022-09-27T15:46:51.4892531Z Running distributed tests for the nccl backend with env init_method
2022-09-27T15:46:51.4899450Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:46:51.489566]
2022-09-27T15:46:51.4904493Z Running distributed tests for the gloo backend with env init_method
2022-09-27T15:46:51.4911136Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:46:51.490765]
2022-09-27T15:46:51.4915978Z Running distributed tests for the ucc backend with env init_method
2022-09-27T15:46:51.4923562Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:46:51.491915]
2022-09-27T15:46:51.4937160Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 15:46:51.493309]
2022-09-27T15:46:53.6490125Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:46:53.6537329Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-09-27T15:46:53.6571712Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6572213Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6572643Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6573070Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6573508Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6573984Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6574578Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6575075Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6575574Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6576124Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6576690Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6577200Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6577738Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6578264Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6578749Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6579221Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6579693Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6580127Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6580540Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6580990Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6581474Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6581956Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6582350Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6582758Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6583193Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6583601Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6584022Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6584449Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6584865Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6585234Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6585629Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6586041Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6586531Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6586939Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6587351Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6587761Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6588184Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6588586Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6589005Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6589430Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6589882Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6590332Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6590752Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6591185Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6591621Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6592122Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6592524Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6592971Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6593417Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6593815Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6594232Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6594660Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6595078Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6595495Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6595917Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6596338Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6596727Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6597126Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6597532Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6597915Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6598304Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6598681Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6599058Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6599474Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6599893Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6600526Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6600912Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6601301Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6601696Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6602074Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6602476Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6602894Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6603266Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6603646Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6604030Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6604499Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6604894Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6605296Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6605701Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6606071Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6606481Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6606920Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6607363Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6607793Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6608252Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6608718Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6609155Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6609598Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6610108Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6610531Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6610975Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6611428Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6611892Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6612338Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6612806Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6613263Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6613672Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6614077Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6614463Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6614832Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6615186Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6615569Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6615968Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6616343Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6616730Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6617135Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6617530Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6617934Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6618337Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6618746Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6619154Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6619572Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6619990Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6620390Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6620809Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6621241Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6621727Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6622136Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6622532Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6622914Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6623288Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6623682Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6624077Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6624461Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6624917Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6625435Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6625891Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6626290Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6626712Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6627200Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6627632Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6628105Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6628553Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6628981Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6629405Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6629818Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6630195Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6630577Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6630995Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6631416Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6631835Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6632282Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6632704Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6633124Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6633571Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6634076Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6634646Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6635321Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6636438Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6637558Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6638794Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6639601Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6640715Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6641356Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6641914Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6642416Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6642846Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6643237Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6643642Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6644038Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6644443Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6644876Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6645304Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6645772Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6646343Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6647122Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6647795Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6648224Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6648671Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6649088Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6649511Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6649932Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6650366Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6650783Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6651213Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6651626Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6652019Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6652445Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6652854Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6653517Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6654146Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6654597Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6654984Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6655368Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6655796Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6656218Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6656586Z test_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6656952Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6657327Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6657709Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6658074Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6658453Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6658844Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6659285Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6659664Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6660023Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6660396Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6660797Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6661196Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6661552Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6661904Z test_isend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6662289Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6662701Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6663100Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6663560Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6664015Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6664414Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6664898Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6665336Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6665769Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6666179Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6666604Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6667023Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6667418Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6667836Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6668243Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6668630Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6669101Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6669602Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6670078Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6670513Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6670976Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6671436Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6671851Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6672279Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6672713Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6673159Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6673595Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6674080Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6674595Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6675065Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6675493Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6675893Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6676300Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6676741Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6677144Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6677529Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6677910Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6678305Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6678681Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6679033Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6679415Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6679800Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6680578Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6680997Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6681375Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6681768Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6682138Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6682509Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6682982Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6683346Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6683726Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6684115Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6684511Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6684885Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6685318Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6685694Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6686054Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6686526Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6686979Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6687397Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6687797Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6688202Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6688630Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6689029Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6689430Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6689847Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6690267Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6690690Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6691096Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6691489Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6691897Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6692281Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6692707Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6693144Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.6693588Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7010712Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:46:53.7057321Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-09-27T15:46:53.7091429Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7091923Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7092345Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7092756Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7093183Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7093655Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7094140Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7094618Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7095133Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7095674Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7096232Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7096737Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7097266Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7097781Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7098344Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7098828Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7099293Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7099720Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7100130Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7100575Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7101058Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7101532Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7101923Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7102323Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7102751Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7103155Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7103664Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7104087Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7104478Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7104861Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7105253Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7105654Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7106023Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7106416Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7106824Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7107228Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7107642Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7108047Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7108442Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7108881Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7109328Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7109774Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7110185Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7110607Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7111041Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7111456Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7111876Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7112318Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7112755Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7113151Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7113560Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7113980Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7114388Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7114795Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7115207Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7115711Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7116109Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7116508Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7116910Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7117293Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7117671Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7118043Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7118409Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7118815Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7119217Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7119609Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7119977Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7120558Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7120955Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7121415Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7121812Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7122222Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7122593Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7122967Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7123354Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7124086Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7124771Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7125187Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7125581Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7125949Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7126663Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7127436Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7128236Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7129063Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7129888Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7130740Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7131366Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7132182Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7132885Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7133678Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7134385Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7135076Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7135835Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7136608Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7137232Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7138005Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7138832Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7139349Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7139788Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7140477Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7141137Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7141859Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7142444Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7142822Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7143426Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7144167Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7144916Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7145662Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7146412Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7146837Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7147326Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7147747Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7148381Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7149136Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7149882Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7150663Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7151446Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7152244Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7152906Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7153596Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7153976Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7154583Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7154982Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7155361Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7155818Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7156329Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7156785Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7157181Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7157595Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7158032Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7158467Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7159060Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7159591Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7160138Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7160910Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7161340Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7161799Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7162268Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7162840Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7163401Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7163936Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7164450Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7164885Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7165413Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7165960Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7166531Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7167113Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7167849Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7168536Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7169299Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7169926Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7170598Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7171314Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7171984Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7172555Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7173124Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7173724Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7219191Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7219621Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7220039Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7220444Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7220847Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7221284Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7221759Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7222211Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7222628Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7223010Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7223415Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7223835Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7224269Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7224668Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7225062Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7225483Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7226053Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7226477Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7226864Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7227259Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7227656Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7228031Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7228431Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7228880Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7229287Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7229648Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7230030Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7230437Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7230824Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7231281Z test_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7231623Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7231964Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7232330Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7232697Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7233052Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7233439Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7233816Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7234173Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7234509Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7234879Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7235270Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7235648Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7236012Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7236348Z test_isend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7236713Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7237116Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7237525Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7237953Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7238404Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7238829Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7239249Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7239668Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7240106Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7240883Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7241278Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7241673Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7242074Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7242476Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7242855Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7243259Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7243789Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7244272Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7244733Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7245165Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7245619Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7246049Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7246475Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7246896Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7247309Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7247740Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7248183Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7248664Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7249225Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7249700Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7250124Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7250504Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7250896Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7251290Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7251657Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7252033Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7252421Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7252811Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7253171Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7253521Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7253896Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7254259Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7254642Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7255019Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7255373Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7255755Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7256134Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7256495Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7256839Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7257204Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7257582Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7257950Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7258345Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7258725Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7259091Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7259463Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7259842Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7260266Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7260749Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7261194Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7261593Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7261985Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7262414Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7262826Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7263206Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7263620Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7264057Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7264468Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7264856Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7265265Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7265668Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7266036Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7266524Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7266980Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7267423Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7267925Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:46:53.7303054Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-09-27T15:46:53.7336819Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7337332Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7337740Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7338155Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7338602Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7339051Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7339530Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7340029Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7340540Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7341149Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7341706Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7342229Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7342740Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7343252Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7343726Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7344212Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7344663Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7345091Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7345519Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7345946Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7346426Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7346900Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7347304Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7347688Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7348116Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7348538Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7348940Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7349361Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7349774Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7350154Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7350527Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7350929Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7351319Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7351689Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7352092Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7352510Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7352972Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7353384Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7353799Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7354248Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7354675Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7355120Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7355548Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7355960Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7356393Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7356824Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7357244Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7357670Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7358108Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7358580Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7358974Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7359397Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7359819Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7360377Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7360807Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7361223Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7361622Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7362003Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7362406Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7362810Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7363174Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7363549Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7363932Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7364327Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7364735Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7365126Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7365507Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7365874Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7366271Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7366667Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7367046Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7367459Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7367844Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7368201Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7368580Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7368965Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7369365Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7369746Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7370140Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7370591Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7370995Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7371424Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7371865Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7372310Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7372748Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7373201Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7373648Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7374071Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7374504Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7374943Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7375386Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7375894Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7376354Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7376815Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7377253Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7377696Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7378119Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7378509Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7378874Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7379238Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7379606Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7379972Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7380364Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7380750Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7381115Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7381510Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7381917Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7382296Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7382694Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7383096Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7383525Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7383927Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7384345Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7384760Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7385156Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7385587Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7386013Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7386467Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7386846Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7387218Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7387665Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7388048Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7388438Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7388836Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7389271Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7389783Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7390238Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7390648Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7391050Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7391485Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7391940Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7392391Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7392829Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7393323Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7393763Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7394155Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7394529Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7394917Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7395312Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7395727Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7396163Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7396589Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7397009Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7397425Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7397887Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7398369Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7398931Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7399545Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7400406Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7401019Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7401629Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7402230Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7402831Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7403417Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7403957Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7404526Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7404986Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7405361Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7405759Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7406172Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7406552Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7406974Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7407412Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7407875Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7408330Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7408749Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7409143Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7409611Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7410046Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7410473Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7410871Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7411286Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7411715Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7412144Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7412547Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7412956Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7413365Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7413768Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7414170Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7414590Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7415052Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7415464Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7415848Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7416247Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7416650Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7417071Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7417450Z test_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7417794Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7418165Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7418543Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7418920Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7419275Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7419662Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7420048Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7420395Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7420751Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7421132Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7421512Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7421981Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7422360Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7422696Z test_isend (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7423077Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7423487Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7423902Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7424343Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7424793Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7425169Z <unittest.suite.TestSuite tests=[]>
2022-09-27T15:46:53.7425522Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7425941Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7426378Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7426809Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7427352Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7427780Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7428197Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7428590Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7429004Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7429399Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7429783Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7430247Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7430742Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7431221Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7431659Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7432118Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7432573Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7432988Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7433413Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7433843Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7434281Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7434721Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7435203Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7435716Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7436185Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7436606Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7437006Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7437410Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7437796Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7438185Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7438566Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7438991Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7439395Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7439770Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7440122Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7440733Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7441123Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7441518Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7441879Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7442253Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7442638Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7443006Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7443373Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7443745Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7444106Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7444481Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7444951Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7445346Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7445711Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7446094Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7446469Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7446831Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7447253Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7447696Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7448108Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7448507Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7448915Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7449347Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7449745Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7450141Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7450555Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7450975Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7451387Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7451795Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7452185Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7452592Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7452978Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7453403Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7453842Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:53.7454284Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T15:46:55.3118384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:55.3118888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:55.3120570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:55.3121304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:55.3738121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:55.3738613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:55.3741397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:55.3741890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:55.3769115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:55.3769576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:55.3773703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:55.3774178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:55.8204917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:46:55.8228494Z 
2022-09-27T15:46:55.8228712Z Running tests...
2022-09-27T15:46:55.8229151Z ----------------------------------------------------------------------
2022-09-27T15:46:55.8628895Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:46:55.8652021Z 
2022-09-27T15:46:55.8652319Z Running tests...
2022-09-27T15:46:55.8652749Z ----------------------------------------------------------------------
2022-09-27T15:46:55.8653284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:46:55.8676869Z 
2022-09-27T15:46:55.8677460Z Running tests...
2022-09-27T15:46:55.8678073Z ----------------------------------------------------------------------
2022-09-27T15:46:57.5420885Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ...   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:46:57.5421713Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:46:57.5448756Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:46:57.5645703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 863
2022-09-27T15:46:57.5646782Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 864
2022-09-27T15:46:57.5649452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 865
2022-09-27T15:46:57.5652965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 866
2022-09-27T15:46:57.5654035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 867
2022-09-27T15:46:57.5657777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 868
2022-09-27T15:46:59.1921587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:59.1922116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:59.1922991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:59.1923472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:59.2079621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:59.2080096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:59.2084121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:59.2084610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:59.2212806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:59.2213532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:59.2217430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:59.2217929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:59.2649845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:59.2650329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:59.2653133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:59.2653630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:59.2965530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:59.2966042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:59.2968991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:59.2979754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:59.2980350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:46:59.2980805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:46:59.2984142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:46:59.2984625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:46:59.7377942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:59.7383378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:59.7384389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:46:59.7478656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:46:59.7485684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:59.7885982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:59.8299537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:59.8300322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:46:59.8393208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:46:59.8396004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:59.8400286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:59.8530542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:46:59.8531074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:46:59.8531783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:46:59.8532472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:46:59.8642291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:46:59.9312864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:46:59.9544577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:01.2327186Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:01.2328220Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:01.3849546Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:01.3850588Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:01.3879977Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:01.3881318Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:01.4729091Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:01.4730449Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:01.4820174Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:01.4821079Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:01.5025101Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:01.5026050Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6322150Z [1664293622.631711] [28fd6574525b:865  :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:47:02.6322697Z [1664293622.631711] [28fd6574525b:865  :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:47:02.6393268Z [1664293622.638955] [28fd6574525b:868  :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:47:02.6393753Z [1664293622.638955] [28fd6574525b:868  :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:47:02.6429287Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6429892Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6430824Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6431745Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6563505Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6564115Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6565002Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6566113Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6696559Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6697175Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6698104Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6698996Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6820912Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6821522Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6822421Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6823591Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6943606Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6944471Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.6945290Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.6946136Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.7060339Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.7061206Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.7063378Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.7064220Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.9845732Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.9846371Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.9847330Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.9848260Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.9966110Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.9967091Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:02.9967737Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:02.9968832Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:03.0087477Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:03.0088852Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:03.0090355Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T15:47:03.0091306Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T15:47:03.3777310Z ok (7.555s)
2022-09-27T15:47:03.3777520Z 
2022-09-27T15:47:03.3777918Z ----------------------------------------------------------------------
2022-09-27T15:47:03.3778259Z Ran 1 test in 7.555s
2022-09-27T15:47:03.3778446Z 
2022-09-27T15:47:03.3778544Z OK
2022-09-27T15:47:03.3778685Z 
2022-09-27T15:47:03.3778819Z Generating XML reports...
2022-09-27T15:47:03.3816402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154655.xml
2022-09-27T15:47:03.4779922Z ok (7.610s)
2022-09-27T15:47:03.4780279Z 
2022-09-27T15:47:03.4780920Z ----------------------------------------------------------------------
2022-09-27T15:47:03.4781507Z Ran 1 test in 7.610s
2022-09-27T15:47:03.4781773Z 
2022-09-27T15:47:03.4781935Z OK
2022-09-27T15:47:03.4782167Z 
2022-09-27T15:47:03.4782395Z Generating XML reports...
2022-09-27T15:47:03.4816612Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154655.xml
2022-09-27T15:47:03.6784241Z ok (7.813s)
2022-09-27T15:47:03.6784634Z 
2022-09-27T15:47:03.6785260Z ----------------------------------------------------------------------
2022-09-27T15:47:03.6785878Z Ran 1 test in 7.813s
2022-09-27T15:47:03.6786168Z 
2022-09-27T15:47:03.6786330Z OK
2022-09-27T15:47:03.6786571Z 
2022-09-27T15:47:03.6786779Z Generating XML reports...
2022-09-27T15:47:03.6821464Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154655.xml
2022-09-27T15:47:05.6679609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:05.6681344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:05.6682546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:05.6683451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:05.6886077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:05.6887027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:05.6890119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:05.6891074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:05.8655523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:05.8656520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:05.8658301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:05.8659212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:06.1583004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:06.1605725Z 
2022-09-27T15:47:06.1606262Z Running tests...
2022-09-27T15:47:06.1607084Z ----------------------------------------------------------------------
2022-09-27T15:47:06.1643035Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-09-27T15:47:06.1643421Z 
2022-09-27T15:47:06.1643718Z ----------------------------------------------------------------------
2022-09-27T15:47:06.1644114Z Ran 1 test in 0.004s
2022-09-27T15:47:06.1644430Z 
2022-09-27T15:47:06.1644586Z OK (skipped=1)
2022-09-27T15:47:06.1644755Z 
2022-09-27T15:47:06.1644883Z Generating XML reports...
2022-09-27T15:47:06.1677767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154706.xml
2022-09-27T15:47:06.1847704Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:47:06.1871655Z 
2022-09-27T15:47:06.1871907Z Running tests...
2022-09-27T15:47:06.1872606Z ----------------------------------------------------------------------
2022-09-27T15:47:06.1910694Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-09-27T15:47:06.1911035Z 
2022-09-27T15:47:06.1911305Z ----------------------------------------------------------------------
2022-09-27T15:47:06.1912144Z Ran 1 test in 0.004s
2022-09-27T15:47:06.1912313Z 
2022-09-27T15:47:06.1912424Z OK (skipped=1)
2022-09-27T15:47:06.1912583Z 
2022-09-27T15:47:06.1912693Z Generating XML reports...
2022-09-27T15:47:06.1947081Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154706.xml
2022-09-27T15:47:06.3581742Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:47:06.3605341Z 
2022-09-27T15:47:06.3606100Z Running tests...
2022-09-27T15:47:06.3606662Z ----------------------------------------------------------------------
2022-09-27T15:47:06.3648036Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-09-27T15:47:06.3648376Z 
2022-09-27T15:47:06.3648663Z ----------------------------------------------------------------------
2022-09-27T15:47:06.3649201Z Ran 1 test in 0.004s
2022-09-27T15:47:06.3649442Z 
2022-09-27T15:47:06.3649554Z OK (skipped=1)
2022-09-27T15:47:06.3649700Z 
2022-09-27T15:47:06.3649829Z Generating XML reports...
2022-09-27T15:47:06.3684468Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154706.xml
2022-09-27T15:47:08.1818645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:08.1819142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:08.1822080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:08.1822579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:08.2248863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:08.2249332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:08.2253301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:08.2253770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:08.4047332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:08.4047810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:08.4049711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:08.4050172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:08.6636308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:08.6658991Z 
2022-09-27T15:47:08.6659294Z Running tests...
2022-09-27T15:47:08.6660013Z ----------------------------------------------------------------------
2022-09-27T15:47:08.7198057Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:47:08.7222314Z 
2022-09-27T15:47:08.7222440Z Running tests...
2022-09-27T15:47:08.7222971Z ----------------------------------------------------------------------
2022-09-27T15:47:08.9097496Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:47:08.9121229Z 
2022-09-27T15:47:08.9121435Z Running tests...
2022-09-27T15:47:08.9122061Z ----------------------------------------------------------------------
2022-09-27T15:47:10.3353460Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:10.3448830Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:10.3545511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1304
2022-09-27T15:47:10.3552122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1305
2022-09-27T15:47:10.3644821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1307
2022-09-27T15:47:10.3652418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1308
2022-09-27T15:47:10.5012915Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:10.5214235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1310
2022-09-27T15:47:10.5220156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1311
2022-09-27T15:47:12.0634033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:12.0634555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:12.0635886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:12.0636383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:12.0711888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:12.0712345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:12.0715743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:12.0716221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:12.0911847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:12.0912297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:12.0915853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:12.0916341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:12.1493280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:12.1493752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:12.1494944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:12.1495415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:12.2543867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:12.2544343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:12.2546881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:12.2547389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:12.2811659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:12.2812108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:12.2815283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:12.2815762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:12.5815224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:12.6202474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:12.6203304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:12.6219822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:12.6227278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:12.6818094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:12.6818596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:12.6819303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:12.6819993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:12.6826110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:12.6826786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:12.7216450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:12.8022959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:12.8080440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:12.8081607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:12.8125246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:12.8131817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:12.9095380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:13.1629353Z ok (4.497s)
2022-09-27T15:47:13.1629580Z 
2022-09-27T15:47:13.1630195Z ----------------------------------------------------------------------
2022-09-27T15:47:13.1630601Z Ran 1 test in 4.497s
2022-09-27T15:47:13.1630755Z 
2022-09-27T15:47:13.1630849Z OK
2022-09-27T15:47:13.1630989Z 
2022-09-27T15:47:13.1631121Z Generating XML reports...
2022-09-27T15:47:13.1680557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154708.xml
2022-09-27T15:47:13.1716357Z ok (4.449s)
2022-09-27T15:47:13.1716703Z 
2022-09-27T15:47:13.1717300Z ----------------------------------------------------------------------
2022-09-27T15:47:13.1717703Z Ran 1 test in 4.449s
2022-09-27T15:47:13.1717869Z 
2022-09-27T15:47:13.1717965Z OK
2022-09-27T15:47:13.1718103Z 
2022-09-27T15:47:13.1718235Z Generating XML reports...
2022-09-27T15:47:13.1755618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154708.xml
2022-09-27T15:47:13.3290525Z ok (4.416s)
2022-09-27T15:47:13.3290744Z 
2022-09-27T15:47:13.3291437Z ----------------------------------------------------------------------
2022-09-27T15:47:13.3292070Z Ran 1 test in 4.417s
2022-09-27T15:47:13.3292260Z 
2022-09-27T15:47:13.3292355Z OK
2022-09-27T15:47:13.3292500Z 
2022-09-27T15:47:13.3292635Z Generating XML reports...
2022-09-27T15:47:13.3328473Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154708.xml
2022-09-27T15:47:15.3888455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:15.3889005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:15.3890254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:15.3890761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:15.4938774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:15.4939606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:15.4941954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:15.4942440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:15.5868696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:15.5869227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:15.5870962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:15.5871470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:15.8732596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:47:15.8754688Z 
2022-09-27T15:47:15.8755108Z Running tests...
2022-09-27T15:47:15.8755559Z ----------------------------------------------------------------------
2022-09-27T15:47:15.9864965Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:47:15.9888399Z 
2022-09-27T15:47:15.9888835Z Running tests...
2022-09-27T15:47:15.9889375Z ----------------------------------------------------------------------
2022-09-27T15:47:16.0692547Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:47:16.0715373Z 
2022-09-27T15:47:16.0715730Z Running tests...
2022-09-27T15:47:16.0716180Z ----------------------------------------------------------------------
2022-09-27T15:47:17.4996637Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:17.5196081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1623
2022-09-27T15:47:17.5202376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1624
2022-09-27T15:47:17.5935384Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:17.6136471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1626
2022-09-27T15:47:17.6143271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1627
2022-09-27T15:47:17.6418220Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:47:17.6615341Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1629
2022-09-27T15:47:17.6620472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1630
2022-09-27T15:47:19.3029539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.3030075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.3030980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.3031470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.3033968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.3034425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.3038687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.3039168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.3831651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.3832129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.3833807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.3834535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.3943106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.3943624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.3945077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.3945554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.4059346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.4059862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.4062916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.4063404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.4591661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:47:19.4592149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:47:19.4595077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:47:19.4595556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:47:19.8243929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:19.8244841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:19.8245889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:19.8344739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:19.8351671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:19.9194431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:19.9258634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:19.9376925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:19.9377427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:47:19.9378423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:19.9379143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:19.9386613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:19.9388720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:20.0144318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:47:20.0145123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:20.0208181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:47:20.0215497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:47:20.1159776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:47:21.2837857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69b5yucm
2022-09-27T15:47:21.2838991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69b5yucm/_remote_module_non_scriptable.py
2022-09-27T15:47:21.4129789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplq7lzfgd
2022-09-27T15:47:21.4130600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplq7lzfgd/_remote_module_non_scriptable.py
2022-09-27T15:47:21.4756887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr79wa4ye
2022-09-27T15:47:21.4757784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr79wa4ye/_remote_module_non_scriptable.py
2022-09-27T15:47:21.4877034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w0c56vm
2022-09-27T15:47:21.4878931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w0c56vm/_remote_module_non_scriptable.py
2022-09-27T15:47:21.5279926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7dymdnt
2022-09-27T15:47:21.5281220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7dymdnt/_remote_module_non_scriptable.py
2022-09-27T15:47:21.5670268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0rz36_h
2022-09-27T15:47:21.5671384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0rz36_h/_remote_module_non_scriptable.py
2022-09-27T15:47:21.5956629Z [1664293641.595027] [28fd6574525b:1629 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:47:21.5957593Z [1664293641.595027] [28fd6574525b:1629 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:47:21.5958645Z [1664293641.595498] [28fd6574525b:1630 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:47:21.5959138Z [1664293641.595498] [28fd6574525b:1630 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:47:22.0177678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.0178234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.0377681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.0378206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.0708094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.0708595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.1119146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.1119687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.1313188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.1313704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.2576542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.2577061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.2778580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.2779067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.3999640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.4000448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.4169138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.4169887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.4199468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.4206360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.5497527Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T15:47:22.5498021Z   warnings.warn(
2022-09-27T15:47:22.5498605Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T15:47:22.5499040Z   warnings.warn(
2022-09-27T15:47:22.5512526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.5515346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.5692819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.5693362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.6923135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.6928959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.7108515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.7113716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.8358796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.8365281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.8473699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.8474726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.8567199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.8571413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.9871993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:47:22.9876070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:22.9849272Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-09-27T15:48:22.9849709Z Traceback (most recent call last):
2022-09-27T15:48:22.9851424Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T15:48:22.9851851Z     getattr(self, test_name)()
2022-09-27T15:48:22.9856168Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T15:48:22.9856595Z     fn()
2022-09-27T15:48:22.9857075Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T15:48:22.9857474Z     return func(*args, **kwargs)
2022-09-27T15:48:22.9858063Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4843, in test_DistributedDataParallel
2022-09-27T15:48:22.9858558Z     self._test_DistributedDataParallel(
2022-09-27T15:48:22.9859158Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4005, in _test_DistributedDataParallel
2022-09-27T15:48:22.9859672Z     model_DDP = nn.parallel.DistributedDataParallel(
2022-09-27T15:48:22.9860226Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 655, in __init__
2022-09-27T15:48:22.9860897Z     _verify_param_shape_across_processes(self.process_group, parameters)
2022-09-27T15:48:22.9861947Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/utils.py", line 112, in _verify_param_shape_across_processes
2022-09-27T15:48:22.9862982Z     return dist._verify_params_across_processes(process_group, tensors, logger)
2022-09-27T15:48:22.9863976Z RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:136] Timed out waiting 60000ms for send operation to complete
2022-09-27T15:48:22.9864435Z  exiting process 1 with exit code: 10
2022-09-27T15:48:23.0130521Z [1664293703.012610] [28fd6574525b:1630 :1]    ucc_schedule.h:182  UCC  WARN  timeout 60 sec has expired on req 0x55625119d2c0, seq_num 65, TL_NCCL, team_id 1, size 2, rank 1, ctx_rank 1: Allgather Cuda inplace=0 bytes=16
2022-09-27T15:48:23.4215708Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-09-27T15:48:23.4232344Z ERROR (67.434s)
2022-09-27T15:48:23.4233188Z     test_DistributedDataParallel errored - num_retries_left: 3
2022-09-27T15:48:23.4233597Z Traceback (most recent call last):
2022-09-27T15:48:23.4234171Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 501, in wrapper
2022-09-27T15:48:23.4234556Z     self._join_processes(fn)
2022-09-27T15:48:23.4235097Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 724, in _join_processes
2022-09-27T15:48:23.4235530Z     self._check_return_codes(elapsed_time)
2022-09-27T15:48:23.4236069Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 769, in _check_return_codes
2022-09-27T15:48:23.4236488Z     raise RuntimeError(error)
2022-09-27T15:48:23.4236836Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-09-27T15:48:23.4237187Z Traceback (most recent call last):
2022-09-27T15:48:23.4237693Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T15:48:23.4238098Z     getattr(self, test_name)()
2022-09-27T15:48:23.4238617Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T15:48:23.4238968Z     fn()
2022-09-27T15:48:23.4239454Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T15:48:23.4239853Z     return func(*args, **kwargs)
2022-09-27T15:48:23.4241096Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4843, in test_DistributedDataParallel
2022-09-27T15:48:23.4241557Z     self._test_DistributedDataParallel(
2022-09-27T15:48:23.4242377Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4005, in _test_DistributedDataParallel
2022-09-27T15:48:23.4242908Z     model_DDP = nn.parallel.DistributedDataParallel(
2022-09-27T15:48:23.4243461Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 655, in __init__
2022-09-27T15:48:23.4243916Z     _verify_param_shape_across_processes(self.process_group, parameters)
2022-09-27T15:48:23.4244527Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/utils.py", line 112, in _verify_param_shape_across_processes
2022-09-27T15:48:23.4245012Z     return dist._verify_params_across_processes(process_group, tensors, logger)
2022-09-27T15:48:23.4245527Z RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:136] Timed out waiting 60000ms for send operation to complete
2022-09-27T15:48:23.4245855Z 
2022-09-27T15:48:23.4245873Z 
2022-09-27T15:48:23.4245893Z 
2022-09-27T15:48:23.4275932Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1872
2022-09-27T15:48:23.4282847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1873
2022-09-27T15:48:28.5335688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:28.5336217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:28.5338435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:28.5338907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:28.5393085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:28.5393560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:28.5397626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:28.5398100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:29.8252564Z [E ProcessGroupNCCL.cpp:821] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=5, OpType=ALLREDUCE, Timeout(ms)=60000) ran for 66800 milliseconds before timing out.
2022-09-27T15:48:30.1265866Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T15:48:30.1266337Z   warnings.warn(
2022-09-27T15:48:30.1267257Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T15:48:30.1267693Z   warnings.warn(
2022-09-27T15:48:30.1343871Z [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
2022-09-27T15:48:30.1387044Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-09-27T15:48:30.1387486Z Traceback (most recent call last):
2022-09-27T15:48:30.1388437Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T15:48:30.1389080Z     getattr(self, test_name)()
2022-09-27T15:48:30.1389626Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T15:48:30.1390003Z     fn()
2022-09-27T15:48:30.1390488Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T15:48:30.1390865Z     return func(*args, **kwargs)
2022-09-27T15:48:30.1391459Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4816, in test_DistributedDataParallel
2022-09-27T15:48:30.1392119Z     self._test_DistributedDataParallel(
2022-09-27T15:48:30.1392764Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4030, in _test_DistributedDataParallel
2022-09-27T15:48:30.1393185Z     self._test_DDP_niter(
2022-09-27T15:48:30.1393737Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3958, in _test_DDP_niter
2022-09-27T15:48:30.1394149Z     self._assert_equal_param(
2022-09-27T15:48:30.1394690Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3913, in _assert_equal_param
2022-09-27T15:48:30.1395126Z     self.assertEqual(p_gpu, p_DDP)
2022-09-27T15:48:30.1395650Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2435, in assertEqual
2022-09-27T15:48:30.1396021Z     assert_equal(
2022-09-27T15:48:30.1396480Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1093, in assert_equal
2022-09-27T15:48:30.1396863Z     raise error_metas[0].to_error(msg)
2022-09-27T15:48:30.1397284Z AssertionError: Tensor-likes are not close!
2022-09-27T15:48:30.1397609Z 
2022-09-27T15:48:30.1397726Z Mismatched elements: 13 / 20 (65.0%)
2022-09-27T15:48:30.1398184Z Greatest absolute difference: 0.0019322186708450317 at index (5, 0) (up to 1e-05 allowed)
2022-09-27T15:48:30.1398684Z Greatest relative difference: 0.008191716062457405 at index (5, 0) (up to 1.3e-06 allowed)
2022-09-27T15:48:30.1399034Z  exiting process 0 with exit code: 10
2022-09-27T15:48:30.1416085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.1418229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.2825902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.2826440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.4234054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.4234595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.4458432Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-09-27T15:48:30.4459154Z Traceback (most recent call last):
2022-09-27T15:48:30.4460081Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T15:48:30.4460494Z     getattr(self, test_name)()
2022-09-27T15:48:30.4461020Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T15:48:30.4461381Z     fn()
2022-09-27T15:48:30.4461865Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T15:48:30.4462261Z     return func(*args, **kwargs)
2022-09-27T15:48:30.4462851Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4816, in test_DistributedDataParallel
2022-09-27T15:48:30.4463328Z     self._test_DistributedDataParallel(
2022-09-27T15:48:30.4463948Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4030, in _test_DistributedDataParallel
2022-09-27T15:48:30.4464387Z     self._test_DDP_niter(
2022-09-27T15:48:30.4464914Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3958, in _test_DDP_niter
2022-09-27T15:48:30.4465341Z     self._assert_equal_param(
2022-09-27T15:48:30.4465896Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3913, in _assert_equal_param
2022-09-27T15:48:30.4466334Z     self.assertEqual(p_gpu, p_DDP)
2022-09-27T15:48:30.4467020Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2435, in assertEqual
2022-09-27T15:48:30.4467409Z     assert_equal(
2022-09-27T15:48:30.4467886Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1093, in assert_equal
2022-09-27T15:48:30.4468261Z     raise error_metas[0].to_error(msg)
2022-09-27T15:48:30.4468630Z AssertionError: Tensor-likes are not close!
2022-09-27T15:48:30.4468833Z 
2022-09-27T15:48:30.4468964Z Mismatched elements: 13 / 20 (65.0%)
2022-09-27T15:48:30.4469416Z Greatest absolute difference: 0.0019322186708450317 at index (5, 0) (up to 1e-05 allowed)
2022-09-27T15:48:30.4469901Z Greatest relative difference: 0.008191716062457405 at index (5, 0) (up to 1.3e-06 allowed)
2022-09-27T15:48:30.4470253Z  exiting process 1 with exit code: 10
2022-09-27T15:48:30.5607101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.5607633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.5785888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.5786444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.6066539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.6070655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.6352802Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-09-27T15:48:30.6366575Z ERROR (74.761s)
2022-09-27T15:48:30.6367290Z     test_DistributedDataParallel errored - num_retries_left: 3
2022-09-27T15:48:30.6367814Z Traceback (most recent call last):
2022-09-27T15:48:30.6368861Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 501, in wrapper
2022-09-27T15:48:30.6369522Z     self._join_processes(fn)
2022-09-27T15:48:30.6370088Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 724, in _join_processes
2022-09-27T15:48:30.6370521Z     self._check_return_codes(elapsed_time)
2022-09-27T15:48:30.6371062Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 769, in _check_return_codes
2022-09-27T15:48:30.6371492Z     raise RuntimeError(error)
2022-09-27T15:48:30.6371834Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-09-27T15:48:30.6372160Z Traceback (most recent call last):
2022-09-27T15:48:30.6372686Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T15:48:30.6373080Z     getattr(self, test_name)()
2022-09-27T15:48:30.6373574Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T15:48:30.6373949Z     fn()
2022-09-27T15:48:30.6374733Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T15:48:30.6375137Z     return func(*args, **kwargs)
2022-09-27T15:48:30.6375712Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4816, in test_DistributedDataParallel
2022-09-27T15:48:30.6376188Z     self._test_DistributedDataParallel(
2022-09-27T15:48:30.6376804Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4030, in _test_DistributedDataParallel
2022-09-27T15:48:30.6377237Z     self._test_DDP_niter(
2022-09-27T15:48:30.6377763Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3958, in _test_DDP_niter
2022-09-27T15:48:30.6378346Z     self._assert_equal_param(
2022-09-27T15:48:30.6378979Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3913, in _assert_equal_param
2022-09-27T15:48:30.6379618Z     self.assertEqual(p_gpu, p_DDP)
2022-09-27T15:48:30.6380317Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2435, in assertEqual
2022-09-27T15:48:30.6380697Z     assert_equal(
2022-09-27T15:48:30.6381406Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1093, in assert_equal
2022-09-27T15:48:30.6381782Z     raise error_metas[0].to_error(msg)
2022-09-27T15:48:30.6382160Z AssertionError: Tensor-likes are not close!
2022-09-27T15:48:30.6382366Z 
2022-09-27T15:48:30.6382499Z Mismatched elements: 13 / 20 (65.0%)
2022-09-27T15:48:30.6382933Z Greatest absolute difference: 0.0019322186708450317 at index (5, 0) (up to 1e-05 allowed)
2022-09-27T15:48:30.6383437Z Greatest relative difference: 0.008191716062457405 at index (5, 0) (up to 1.3e-06 allowed)
2022-09-27T15:48:30.6383671Z 
2022-09-27T15:48:30.6383691Z 
2022-09-27T15:48:30.6383710Z 
2022-09-27T15:48:30.6406674Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1940
2022-09-27T15:48:30.6413522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1941
2022-09-27T15:48:30.7079088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:30.7079636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:30.7080802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:30.7081518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:30.7187218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:30.7443014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.7447339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.7629030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.7637509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.8091665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:30.8871816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.8879552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.9068287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:30.9075180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.0369014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.0375422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.1806738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.1813613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.2206691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.2207277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.2601044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.2601616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:31.9847614Z ok (75.913s)
2022-09-27T15:48:31.9848206Z 
2022-09-27T15:48:31.9848938Z ----------------------------------------------------------------------
2022-09-27T15:48:31.9849294Z Ran 1 test in 75.913s
2022-09-27T15:48:31.9849715Z 
2022-09-27T15:48:31.9849822Z OK
2022-09-27T15:48:31.9849959Z 
2022-09-27T15:48:31.9850078Z Generating XML reports...
2022-09-27T15:48:31.9884917Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154716.xml
2022-09-27T15:48:32.3255296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:32.3255797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:32.3258242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:32.3258745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:32.3261218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpci0cgcm6
2022-09-27T15:48:32.3264342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpci0cgcm6/_remote_module_non_scriptable.py
2022-09-27T15:48:32.3804656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:32.3805458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:32.3806909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:32.3807390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:32.4361056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdx8l7ef
2022-09-27T15:48:32.4361860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdx8l7ef/_remote_module_non_scriptable.py
2022-09-27T15:48:32.8615629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:32.9195097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:32.9196072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:32.9223814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:32.9231389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:32.9925628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:32.9926551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.0165899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.0166833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.0207375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:33.0520056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.0520894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.0937847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.0938436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.1182084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.1193581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.2604619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.2605181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.2804193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.2804728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.3123077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.3123809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.4432393Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T15:48:33.4432872Z   warnings.warn(
2022-09-27T15:48:33.4433445Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T15:48:33.4433876Z   warnings.warn(
2022-09-27T15:48:33.4600548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.4601317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.5058089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.5059293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.5520723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.5521295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.6943510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.6944066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.7138873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.7139403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.8474831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.8475378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.9904377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:33.9904972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.0105576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.0106112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.0370068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.0370627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.0567046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.0567600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.1654582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:34.1655112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:34.1657250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:34.1657747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:34.1877580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.1878125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.3018414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64pkgl_l
2022-09-27T15:48:34.3019624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64pkgl_l/_remote_module_non_scriptable.py
2022-09-27T15:48:34.3331665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.3332186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.3521879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylko3jng
2022-09-27T15:48:34.3524436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylko3jng/_remote_module_non_scriptable.py
2022-09-27T15:48:34.4754694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.4755212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.5166603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.5167130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:34.6875027Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:48:34.6899973Z 
2022-09-27T15:48:34.6900389Z Running tests...
2022-09-27T15:48:34.6901234Z ----------------------------------------------------------------------
2022-09-27T15:48:35.1497839Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... ok (11.726s)
2022-09-27T15:48:35.1498185Z 
2022-09-27T15:48:35.1498563Z ----------------------------------------------------------------------
2022-09-27T15:48:35.1498886Z Ran 2 tests in 79.161s
2022-09-27T15:48:35.1499051Z 
2022-09-27T15:48:35.1499195Z OK (skipped=1, expected failures=1)
2022-09-27T15:48:35.1499384Z 
2022-09-27T15:48:35.1499510Z Generating XML reports...
2022-09-27T15:48:35.1539150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154715.xml
2022-09-27T15:48:36.0065116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.0065698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.0247045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.0247565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.0537869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.0538387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.1910725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.1911237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.2087933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.2094648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.3383867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.3385190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.3740820Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:36.3755933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.3777543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.4167291Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2107
2022-09-27T15:48:36.4206056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2108
2022-09-27T15:48:36.5140802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.5141363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.6399671Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T15:48:36.6400451Z   warnings.warn(
2022-09-27T15:48:36.6401085Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T15:48:36.6401514Z   warnings.warn(
2022-09-27T15:48:36.6544111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.6544604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.7938703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.7939245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.9333946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:36.9334497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.0741501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.0768117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.1152921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.1154614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.2477548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.2480133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.3545977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:37.3546510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:37.3548293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:37.3548773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:37.3838546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.3839074Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.4019174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.4019677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.4243329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.4243868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.4428231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.4428761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.4705435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.4706006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.6118850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.6119393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.7537156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.7537694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.8551525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:37.8575747Z 
2022-09-27T15:48:37.8575908Z Running tests...
2022-09-27T15:48:37.8576348Z ----------------------------------------------------------------------
2022-09-27T15:48:37.8957313Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:37.8957920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:38.0841474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:38.0842006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:38.0843498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:38.0843980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:38.1051206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:38.1051689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:38.1053719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:38.1054207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:38.6130870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:38.6308836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:38.6309647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:38.6333987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:38.6341263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:38.6448518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvebo1r4q
2022-09-27T15:48:38.6451428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvebo1r4q/_remote_module_non_scriptable.py
2022-09-27T15:48:38.6579663Z ok (8.021s)
2022-09-27T15:48:38.6580931Z 
2022-09-27T15:48:38.6581315Z ----------------------------------------------------------------------
2022-09-27T15:48:38.6581641Z Ran 2 tests in 82.783s
2022-09-27T15:48:38.6581813Z 
2022-09-27T15:48:38.6581956Z OK (skipped=1, expected failures=1)
2022-09-27T15:48:38.6582143Z 
2022-09-27T15:48:38.6582269Z Generating XML reports...
2022-09-27T15:48:38.6621208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154715.xml
2022-09-27T15:48:38.7322930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:38.7430979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppplp77pb
2022-09-27T15:48:38.7433014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppplp77pb/_remote_module_non_scriptable.py
2022-09-27T15:48:39.5900164Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:39.6127742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2181
2022-09-27T15:48:39.6132790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2182
2022-09-27T15:48:40.1455506Z [1664293720.144992] [28fd6574525b:2107 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:48:40.1456013Z [1664293720.144992] [28fd6574525b:2107 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:48:40.1853424Z [1664293720.184847] [28fd6574525b:2108 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:48:40.1854164Z [1664293720.184847] [28fd6574525b:2108 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:48:40.6131278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:40.6131826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:40.9070577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:40.9071099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:40.9072310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:40.9072802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:41.1320968Z ok (6.442s)
2022-09-27T15:48:41.1321471Z 
2022-09-27T15:48:41.1321885Z ----------------------------------------------------------------------
2022-09-27T15:48:41.1322208Z Ran 1 test in 6.442s
2022-09-27T15:48:41.1322374Z 
2022-09-27T15:48:41.1322470Z OK
2022-09-27T15:48:41.1322611Z 
2022-09-27T15:48:41.1322746Z Generating XML reports...
2022-09-27T15:48:41.1379046Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154834.xml
2022-09-27T15:48:41.3701288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:41.3701939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:41.3702915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:41.3703401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:41.3757650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:41.3758139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:41.3761704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:41.3762205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:41.4067997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:48:41.4090752Z 
2022-09-27T15:48:41.4091356Z Running tests...
2022-09-27T15:48:41.4091951Z ----------------------------------------------------------------------
2022-09-27T15:48:41.4124199Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.003s)
2022-09-27T15:48:41.4124968Z 
2022-09-27T15:48:41.4125527Z ----------------------------------------------------------------------
2022-09-27T15:48:41.4125886Z Ran 1 test in 0.003s
2022-09-27T15:48:41.4126033Z 
2022-09-27T15:48:41.4126160Z OK (skipped=1)
2022-09-27T15:48:41.4126316Z 
2022-09-27T15:48:41.4126442Z Generating XML reports...
2022-09-27T15:48:41.4161245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154841.xml
2022-09-27T15:48:41.9330013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:41.9330593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:41.9331362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:41.9332063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:41.9338357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:41.9339090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:41.9454391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_wy5885
2022-09-27T15:48:41.9457430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_wy5885/_remote_module_non_scriptable.py
2022-09-27T15:48:41.9464283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfodkxrzv
2022-09-27T15:48:41.9467457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfodkxrzv/_remote_module_non_scriptable.py
2022-09-27T15:48:41.9684989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:41.9685529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:42.3210469Z ok (4.462s)
2022-09-27T15:48:42.3210884Z 
2022-09-27T15:48:42.3211542Z ----------------------------------------------------------------------
2022-09-27T15:48:42.3212198Z Ran 1 test in 4.462s
2022-09-27T15:48:42.3212501Z 
2022-09-27T15:48:42.3212715Z OK
2022-09-27T15:48:42.3212949Z 
2022-09-27T15:48:42.3213202Z Generating XML reports...
2022-09-27T15:48:42.3252384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154837.xml
2022-09-27T15:48:43.2924639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:43.2925139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:43.2927143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:43.2927633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:43.4188440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:43.4188914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:43.4190589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:43.4191073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:43.8703739Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:48:43.8727122Z 
2022-09-27T15:48:43.8727554Z Running tests...
2022-09-27T15:48:43.8728053Z ----------------------------------------------------------------------
2022-09-27T15:48:43.9124325Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:48:43.9148535Z 
2022-09-27T15:48:43.9148790Z Running tests...
2022-09-27T15:48:43.9149231Z ----------------------------------------------------------------------
2022-09-27T15:48:43.9166702Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-09-27T15:48:43.9167157Z 
2022-09-27T15:48:43.9167530Z ----------------------------------------------------------------------
2022-09-27T15:48:43.9167904Z Ran 1 test in 0.002s
2022-09-27T15:48:43.9168052Z 
2022-09-27T15:48:43.9168177Z OK (skipped=1)
2022-09-27T15:48:43.9168335Z 
2022-09-27T15:48:43.9168463Z Generating XML reports...
2022-09-27T15:48:43.9211485Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154843.xml
2022-09-27T15:48:44.4664191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:44.4664696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:44.4666764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:44.4667577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:45.0319480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:45.0344673Z 
2022-09-27T15:48:45.0345243Z Running tests...
2022-09-27T15:48:45.0346169Z ----------------------------------------------------------------------
2022-09-27T15:48:45.5301171Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:45.5508078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2434
2022-09-27T15:48:45.5513661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2435
2022-09-27T15:48:45.8790438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:45.8790948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:45.8791835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:45.8792304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:46.4308202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:48:46.4330134Z 
2022-09-27T15:48:46.4330555Z Running tests...
2022-09-27T15:48:46.4331074Z ----------------------------------------------------------------------
2022-09-27T15:48:46.6744781Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:46.6959056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2501
2022-09-27T15:48:46.6965030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2502
2022-09-27T15:48:47.1746238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:47.1746778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:47.1748339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:47.1748829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:47.1890103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:47.1890587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:47.1893361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:47.1893845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:47.7591752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:47.8025181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:47.8026182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:47.8097612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:47.8104705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:47.8211347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwc4ywncf
2022-09-27T15:48:47.8214001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwc4ywncf/_remote_module_non_scriptable.py
2022-09-27T15:48:47.9037071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:47.9147052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvdo3li5
2022-09-27T15:48:47.9150421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvdo3li5/_remote_module_non_scriptable.py
2022-09-27T15:48:48.0432485Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:48.0635119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2572
2022-09-27T15:48:48.0639963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2573
2022-09-27T15:48:48.3351131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:48.3352136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:48.3372926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:48.3373909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:48.4260123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:48.4261151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:48.4262369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:48.4291418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:49.0803825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:49.0804420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:49.0805789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:49.0905762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:49.0912975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:49.1020171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvih7anz
2022-09-27T15:48:49.1022741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvih7anz/_remote_module_non_scriptable.py
2022-09-27T15:48:49.1819993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:49.1929880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51gdm45w
2022-09-27T15:48:49.1932676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51gdm45w/_remote_module_non_scriptable.py
2022-09-27T15:48:49.2164115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:49.2164633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:49.3393333Z [1664293729.338803] [28fd6574525b:2435 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:48:49.3393886Z [1664293729.338803] [28fd6574525b:2435 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:48:49.3452268Z [1664293729.344816] [28fd6574525b:2434 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:48:49.3452803Z [1664293729.344816] [28fd6574525b:2434 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:48:49.6041049Z ok (4.569s)
2022-09-27T15:48:49.6041262Z 
2022-09-27T15:48:49.6041653Z ----------------------------------------------------------------------
2022-09-27T15:48:49.6041997Z Ran 1 test in 4.570s
2022-09-27T15:48:49.6042167Z 
2022-09-27T15:48:49.6042244Z OK
2022-09-27T15:48:49.6042380Z 
2022-09-27T15:48:49.6042517Z Generating XML reports...
2022-09-27T15:48:49.6079381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154845.xml
2022-09-27T15:48:49.6744577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:49.6745389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:49.7207994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:49.7208657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:49.7210314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:49.7210777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:49.7788543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:49.7789021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:49.7791597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:49.7792077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:50.2517856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:50.3176545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:50.3177410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:50.3226003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:50.3233575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:50.3630053Z ok (6.490s)
2022-09-27T15:48:50.3630270Z 
2022-09-27T15:48:50.3630629Z ----------------------------------------------------------------------
2022-09-27T15:48:50.3630973Z Ran 1 test in 6.490s
2022-09-27T15:48:50.3631140Z 
2022-09-27T15:48:50.3631256Z OK
2022-09-27T15:48:50.3631394Z 
2022-09-27T15:48:50.3631508Z Generating XML reports...
2022-09-27T15:48:50.3668953Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154843.xml
2022-09-27T15:48:50.4190483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:51.7207817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25jb4h4p
2022-09-27T15:48:51.7208438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25jb4h4p/_remote_module_non_scriptable.py
2022-09-27T15:48:51.7973224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:51.7973883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:51.7975257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:51.7975764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:51.8485265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpft8f8et4
2022-09-27T15:48:51.8486571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpft8f8et4/_remote_module_non_scriptable.py
2022-09-27T15:48:52.3001075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:48:52.3023415Z 
2022-09-27T15:48:52.3023924Z Running tests...
2022-09-27T15:48:52.3024413Z ----------------------------------------------------------------------
2022-09-27T15:48:52.5847729Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:52.5848655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:52.5849745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:52.5850255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:53.0994529Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:48:53.1019072Z 
2022-09-27T15:48:53.1019541Z Running tests...
2022-09-27T15:48:53.1020062Z ----------------------------------------------------------------------
2022-09-27T15:48:53.7234064Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.7234719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.7437803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.7438314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.7715326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.7715852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.7920451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.7921180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.8985470Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:53.9181472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2750
2022-09-27T15:48:53.9187157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2751
2022-09-27T15:48:53.9191505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.9193550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.9399103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:53.9399621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:54.7381545Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:54.7580400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2815
2022-09-27T15:48:54.7587331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2816
2022-09-27T15:48:54.7779835Z ok (8.345s)
2022-09-27T15:48:54.7779999Z 
2022-09-27T15:48:54.7780534Z ----------------------------------------------------------------------
2022-09-27T15:48:54.7780939Z Ran 1 test in 8.345s
2022-09-27T15:48:54.7781108Z 
2022-09-27T15:48:54.7781212Z OK
2022-09-27T15:48:54.7781354Z 
2022-09-27T15:48:54.7781469Z Generating XML reports...
2022-09-27T15:48:54.7817937Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154846.xml
2022-09-27T15:48:55.6361792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:55.6362308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:55.6363771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:55.6364263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:55.6621525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:55.6622032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:55.6624097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:55.6624595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:56.2637049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:56.2637626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:56.2638416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:56.2639353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:56.2646440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:56.2647569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:56.4996384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:56.4996981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:56.4998288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:56.4999108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:56.5052269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:56.5053059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:56.5056046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:56.5056866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:56.9426210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:48:56.9426932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:48:56.9429718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:48:56.9430522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:48:57.0311531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:48:57.0529694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:48:57.0530905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:57.0616881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:48:57.0623587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:48:57.1543572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:48:57.4625967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:48:57.4650087Z 
2022-09-27T15:48:57.4650415Z Running tests...
2022-09-27T15:48:57.4650851Z ----------------------------------------------------------------------
2022-09-27T15:48:57.6510208Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvekvsfhq
2022-09-27T15:48:57.6511159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvekvsfhq/_remote_module_non_scriptable.py
2022-09-27T15:48:57.6526547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph629u9_b
2022-09-27T15:48:57.6529473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph629u9_b/_remote_module_non_scriptable.py
2022-09-27T15:48:58.4036610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.4291160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.4291740Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.4292221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.4570459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tke4q0z
2022-09-27T15:48:58.4571576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tke4q0z/_remote_module_non_scriptable.py
2022-09-27T15:48:58.4615999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.4616621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.4862861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.4863416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.5156148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.5156657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.5157357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lbc6k3n
2022-09-27T15:48:58.5160722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lbc6k3n/_remote_module_non_scriptable.py
2022-09-27T15:48:58.5396505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.5397027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:58.5434150Z [1664293738.543077] [28fd6574525b:2816 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:48:58.5434634Z [1664293738.543080] [28fd6574525b:2815 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:48:58.5435119Z [1664293738.543077] [28fd6574525b:2816 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:48:58.5435597Z [1664293738.543080] [28fd6574525b:2815 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:48:59.1091565Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:48:59.1290753Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2945
2022-09-27T15:48:59.1297713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2946
2022-09-27T15:48:59.1302694Z ok (6.828s)
2022-09-27T15:48:59.1303392Z 
2022-09-27T15:48:59.1303925Z ----------------------------------------------------------------------
2022-09-27T15:48:59.1304320Z Ran 1 test in 6.828s
2022-09-27T15:48:59.1304500Z 
2022-09-27T15:48:59.1304595Z OK
2022-09-27T15:48:59.1304735Z 
2022-09-27T15:48:59.1304875Z Generating XML reports...
2022-09-27T15:48:59.1341915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154852.xml
2022-09-27T15:48:59.6623897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.6624465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.6833561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.6835650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.7120078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.7121366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.7327730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.7331197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.8758610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.8759171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.8967424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:48:59.8971595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:00.6719219Z ok (7.570s)
2022-09-27T15:49:00.6719460Z 
2022-09-27T15:49:00.6719835Z ----------------------------------------------------------------------
2022-09-27T15:49:00.6720364Z Ran 1 test in 7.570s
2022-09-27T15:49:00.6720542Z 
2022-09-27T15:49:00.6720642Z OK
2022-09-27T15:49:00.6720782Z 
2022-09-27T15:49:00.6720927Z Generating XML reports...
2022-09-27T15:49:00.6777490Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154853.xml
2022-09-27T15:49:00.8392097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:00.8392683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:00.8395118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:00.8395921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:00.8857303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:00.8857997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:00.8860149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:00.8861026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:01.2832103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:01.2832632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:01.2833693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:01.2834182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:01.3548622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:01.3986865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:01.3987676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:01.4055786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:01.4062469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:01.4999872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:01.7992923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:01.8036534Z 
2022-09-27T15:49:01.8036991Z Running tests...
2022-09-27T15:49:01.8038435Z ----------------------------------------------------------------------
2022-09-27T15:49:02.7327377Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8omf2rhx
2022-09-27T15:49:02.7328662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8omf2rhx/_remote_module_non_scriptable.py
2022-09-27T15:49:02.8405319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:02.8406123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:02.8407806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:02.8408305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:02.9256187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcd2mtbto
2022-09-27T15:49:02.9257311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcd2mtbto/_remote_module_non_scriptable.py
2022-09-27T15:49:03.3813210Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:49:03.3836427Z 
2022-09-27T15:49:03.3836928Z Running tests...
2022-09-27T15:49:03.3837416Z ----------------------------------------------------------------------
2022-09-27T15:49:03.4549503Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:03.4753795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3095
2022-09-27T15:49:03.4758582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3096
2022-09-27T15:49:03.8692118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:03.8692668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:03.8824454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:04.5421626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:04.5422057Z ok (7.077s)
2022-09-27T15:49:04.5422216Z 
2022-09-27T15:49:04.5422619Z ----------------------------------------------------------------------
2022-09-27T15:49:04.5422945Z Ran 1 test in 7.077s
2022-09-27T15:49:04.5423116Z 
2022-09-27T15:49:04.5423214Z OK
2022-09-27T15:49:04.5423355Z 
2022-09-27T15:49:04.5423509Z Generating XML reports...
2022-09-27T15:49:04.5457757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154857.xml
2022-09-27T15:49:05.1090872Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:05.1290844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3170
2022-09-27T15:49:05.1296967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3171
2022-09-27T15:49:05.1421801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:05.1422282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:05.1425600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:05.1426088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:05.1474481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:05.1474946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:05.1478870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:05.1479366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:05.7158246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:05.7162130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:05.7164339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:05.7258389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:05.7266548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:05.8242848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:06.7625111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:06.7625619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:06.7628073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:06.7628580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:06.8099227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:06.8099731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:06.8101411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:06.8101952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:06.8170234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:06.8171044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:06.8174190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:06.8174677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:07.1153950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcktyqr_d
2022-09-27T15:49:07.1155273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcktyqr_d/_remote_module_non_scriptable.py
2022-09-27T15:49:07.1547396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_c4bnbm5
2022-09-27T15:49:07.1548954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_c4bnbm5/_remote_module_non_scriptable.py
2022-09-27T15:49:07.1817549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:07.1818310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:07.1980841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:07.1981525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:07.3182316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:49:07.3206785Z 
2022-09-27T15:49:07.3207030Z Running tests...
2022-09-27T15:49:07.3207650Z ----------------------------------------------------------------------
2022-09-27T15:49:07.3338643Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:07.3394544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:07.3395353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:07.3441460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:07.3448484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:07.4410496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:07.6859344Z ok (5.882s)
2022-09-27T15:49:07.6859553Z 
2022-09-27T15:49:07.6859942Z ----------------------------------------------------------------------
2022-09-27T15:49:07.6860296Z Ran 1 test in 5.882s
2022-09-27T15:49:07.6860468Z 
2022-09-27T15:49:07.6860828Z OK
2022-09-27T15:49:07.6860988Z 
2022-09-27T15:49:07.6861105Z Generating XML reports...
2022-09-27T15:49:07.6897251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154901.xml
2022-09-27T15:49:08.7221528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdm0qn9rj
2022-09-27T15:49:08.7222647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdm0qn9rj/_remote_module_non_scriptable.py
2022-09-27T15:49:08.8069485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkz5hk08c
2022-09-27T15:49:08.8070104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkz5hk08c/_remote_module_non_scriptable.py
2022-09-27T15:49:08.8318140Z [1664293748.831462] [28fd6574525b:3170 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:08.8318665Z [1664293748.831462] [28fd6574525b:3170 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:08.8374244Z [1664293748.837131] [28fd6574525b:3171 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:08.8374977Z [1664293748.837131] [28fd6574525b:3171 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:09.1020944Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:09.1228776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3327
2022-09-27T15:49:09.1233968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3328
2022-09-27T15:49:09.2569639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:09.2570672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:09.2702927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:09.2703918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:09.8411602Z ok (6.457s)
2022-09-27T15:49:09.8412792Z 
2022-09-27T15:49:09.8414901Z ----------------------------------------------------------------------
2022-09-27T15:49:09.8415699Z Ran 1 test in 6.457s
2022-09-27T15:49:09.8416300Z 
2022-09-27T15:49:09.8420029Z OK
2022-09-27T15:49:09.8420345Z 
2022-09-27T15:49:09.8422517Z Generating XML reports...
2022-09-27T15:49:09.8481460Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154903.xml
2022-09-27T15:49:09.9484606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:09.9485110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:09.9487596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:09.9488118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:10.4520729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:10.4544092Z 
2022-09-27T15:49:10.4544326Z Running tests...
2022-09-27T15:49:10.4545541Z ----------------------------------------------------------------------
2022-09-27T15:49:10.7757317Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:10.7758566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:10.7759720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:10.7761000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:10.7807454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:10.7808361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:10.7811788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:10.7812747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:11.3555255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:11.3675289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:11.3676114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:11.3757874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:11.3764645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:11.4690756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:11.9859755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:11.9860249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:11.9862626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:11.9863115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:12.1000146Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:12.1206072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3445
2022-09-27T15:49:12.1211798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3446
2022-09-27T15:49:12.4987096Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:49:12.5011517Z 
2022-09-27T15:49:12.5011666Z Running tests...
2022-09-27T15:49:12.5012154Z ----------------------------------------------------------------------
2022-09-27T15:49:12.7134130Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9elt66fr
2022-09-27T15:49:12.7135340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9elt66fr/_remote_module_non_scriptable.py
2022-09-27T15:49:12.8869754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwie_bcs8
2022-09-27T15:49:12.8870640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwie_bcs8/_remote_module_non_scriptable.py
2022-09-27T15:49:13.7934251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:13.7934785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:13.7935999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:13.7936478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:13.8955409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:13.8955908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:13.8957807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:13.8958460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:13.9529808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:13.9530692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:13.9682011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:13.9682544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:14.2284941Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:14.2486159Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3525
2022-09-27T15:49:14.2490986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3526
2022-09-27T15:49:14.4638204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:14.4638741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:14.4639529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:14.4640501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:14.4647855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:14.4648991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:14.5359157Z ok (7.215s)
2022-09-27T15:49:14.5359379Z 
2022-09-27T15:49:14.5359762Z ----------------------------------------------------------------------
2022-09-27T15:49:14.5360107Z Ran 1 test in 7.215s
2022-09-27T15:49:14.5360579Z 
2022-09-27T15:49:14.5360670Z OK
2022-09-27T15:49:14.5360814Z 
2022-09-27T15:49:14.5360951Z Generating XML reports...
2022-09-27T15:49:14.5402667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154907.xml
2022-09-27T15:49:15.9095398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6p0dbfcs
2022-09-27T15:49:15.9096029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6p0dbfcs/_remote_module_non_scriptable.py
2022-09-27T15:49:15.9572240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1wvfzb4
2022-09-27T15:49:15.9573254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1wvfzb4/_remote_module_non_scriptable.py
2022-09-27T15:49:15.9596807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:15.9597282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:15.9601171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:15.9601670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:15.9609395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:15.9609867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:15.9613734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:15.9614223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:15.9877759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:15.9878302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:16.0068800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:16.0069308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:16.4961447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:16.5108604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:16.5109661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:16.5163940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:16.5170782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:16.5308342Z ok (6.076s)
2022-09-27T15:49:16.5308936Z 
2022-09-27T15:49:16.5309324Z ----------------------------------------------------------------------
2022-09-27T15:49:16.5309663Z Ran 1 test in 6.076s
2022-09-27T15:49:16.5309833Z 
2022-09-27T15:49:16.5309909Z OK
2022-09-27T15:49:16.5310054Z 
2022-09-27T15:49:16.5310190Z Generating XML reports...
2022-09-27T15:49:16.5346626Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154910.xml
2022-09-27T15:49:16.6123647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:16.7092633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:16.7093482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:16.7094949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:16.7095434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:17.2268063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:49:17.2290793Z 
2022-09-27T15:49:17.2291253Z Running tests...
2022-09-27T15:49:17.2291755Z ----------------------------------------------------------------------
2022-09-27T15:49:17.8803584Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrp3v92e
2022-09-27T15:49:17.8805029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrp3v92e/_remote_module_non_scriptable.py
2022-09-27T15:49:18.0297673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmploa52o4j
2022-09-27T15:49:18.0298898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmploa52o4j/_remote_module_non_scriptable.py
2022-09-27T15:49:18.0575406Z [1664293758.057018] [28fd6574525b:3525 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:18.0576373Z [1664293758.057018] [28fd6574525b:3525 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:18.0654456Z [1664293758.065083] [28fd6574525b:3526 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:18.0655497Z [1664293758.065083] [28fd6574525b:3526 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:18.4192420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:18.4193428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:18.4349829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:18.4351560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:18.6649964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:18.6650492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:18.6651604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:18.6652107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:18.8250530Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:18.8450048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3690
2022-09-27T15:49:18.8455206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3691
2022-09-27T15:49:19.0612944Z ok (6.560s)
2022-09-27T15:49:19.0613183Z 
2022-09-27T15:49:19.0613574Z ----------------------------------------------------------------------
2022-09-27T15:49:19.0613923Z Ran 1 test in 6.560s
2022-09-27T15:49:19.0614093Z 
2022-09-27T15:49:19.0614171Z OK
2022-09-27T15:49:19.0614308Z 
2022-09-27T15:49:19.0614449Z Generating XML reports...
2022-09-27T15:49:19.0668374Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154912.xml
2022-09-27T15:49:19.2113084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:19.2135863Z 
2022-09-27T15:49:19.2136110Z Running tests...
2022-09-27T15:49:19.2136581Z ----------------------------------------------------------------------
2022-09-27T15:49:20.5323218Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:20.5325087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:20.5326280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:20.5327248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:20.5772502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:20.5773462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:20.5774671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:20.5775617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:20.8563330Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:20.8755042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3790
2022-09-27T15:49:20.8761381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3791
2022-09-27T15:49:21.0380654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:21.0918874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:21.0920587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:21.0987736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:21.0995219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:21.1933269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:21.2869441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:21.2870428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:21.2871532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:21.2872491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:21.8716975Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:49:21.8740269Z 
2022-09-27T15:49:21.8740729Z Running tests...
2022-09-27T15:49:21.8741597Z ----------------------------------------------------------------------
2022-09-27T15:49:22.4799997Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpru1cgn8c
2022-09-27T15:49:22.4801825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpru1cgn8c/_remote_module_non_scriptable.py
2022-09-27T15:49:22.5370515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:22.5371518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:22.5372702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:22.5373685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:22.5441148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxaycne1
2022-09-27T15:49:22.5443526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxaycne1/_remote_module_non_scriptable.py
2022-09-27T15:49:22.5481132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:22.5482102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:22.5484616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:22.5485568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:23.0921345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:23.0922228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:23.0923470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:23.0924599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:23.0929845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:23.0930622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:23.5754342Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:23.6198004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3883
2022-09-27T15:49:23.6203153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3884
2022-09-27T15:49:23.6665515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:23.6666541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:24.5606980Z ok (7.331s)
2022-09-27T15:49:24.5607205Z 
2022-09-27T15:49:24.5607605Z ----------------------------------------------------------------------
2022-09-27T15:49:24.5607929Z Ran 1 test in 7.331s
2022-09-27T15:49:24.5608118Z 
2022-09-27T15:49:24.5608223Z OK
2022-09-27T15:49:24.5608359Z 
2022-09-27T15:49:24.5608499Z Generating XML reports...
2022-09-27T15:49:24.5645086Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154917.xml
2022-09-27T15:49:24.6138279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqpv6lp9
2022-09-27T15:49:24.6140433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqpv6lp9/_remote_module_non_scriptable.py
2022-09-27T15:49:24.6222108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplffuwieq
2022-09-27T15:49:24.6224518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplffuwieq/_remote_module_non_scriptable.py
2022-09-27T15:49:24.6473425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:24.6474274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:25.2980269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:25.2980785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:25.2982274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:25.2982770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:25.3115338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:25.3116097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:25.3119687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:25.3120560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:25.3880140Z ok (6.174s)
2022-09-27T15:49:25.3880689Z 
2022-09-27T15:49:25.3881235Z ----------------------------------------------------------------------
2022-09-27T15:49:25.3881777Z Ran 1 test in 6.174s
2022-09-27T15:49:25.3882185Z 
2022-09-27T15:49:25.3882284Z OK
2022-09-27T15:49:25.3882421Z 
2022-09-27T15:49:25.3882554Z Generating XML reports...
2022-09-27T15:49:25.3936942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154919.xml
2022-09-27T15:49:25.8226965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:25.9214227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:25.9215057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:25.9240781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:25.9251414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:26.0227418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:26.7642147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:26.7642670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:26.7643730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:26.7644212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:27.2740211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5tnf62a
2022-09-27T15:49:27.2741534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5tnf62a/_remote_module_non_scriptable.py
2022-09-27T15:49:27.2824626Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:49:27.2849707Z 
2022-09-27T15:49:27.2850704Z Running tests...
2022-09-27T15:49:27.2851703Z ----------------------------------------------------------------------
2022-09-27T15:49:27.2873093Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-09-27T15:49:27.2874031Z 
2022-09-27T15:49:27.2874413Z ----------------------------------------------------------------------
2022-09-27T15:49:27.2874752Z Ran 1 test in 0.002s
2022-09-27T15:49:27.2874899Z 
2022-09-27T15:49:27.2875011Z OK (skipped=1)
2022-09-27T15:49:27.2875168Z 
2022-09-27T15:49:27.2875293Z Generating XML reports...
2022-09-27T15:49:27.2909198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154927.xml
2022-09-27T15:49:27.3297424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5bllgdn
2022-09-27T15:49:27.3298616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5bllgdn/_remote_module_non_scriptable.py
2022-09-27T15:49:27.3566809Z [1664293767.356217] [28fd6574525b:3883 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:27.3567691Z [1664293767.356217] [28fd6574525b:3883 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:27.3599736Z [1664293767.359649] [28fd6574525b:3884 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:27.3600984Z [1664293767.359649] [28fd6574525b:3884 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:27.5123567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:27.5124170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:27.5125524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:27.5126560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:27.6984048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:27.6985026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:28.0250905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:28.0273673Z 
2022-09-27T15:49:28.0274049Z Running tests...
2022-09-27T15:49:28.0275044Z ----------------------------------------------------------------------
2022-09-27T15:49:28.0298352Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-09-27T15:49:28.0298798Z 
2022-09-27T15:49:28.0299131Z ----------------------------------------------------------------------
2022-09-27T15:49:28.0299467Z Ran 1 test in 0.002s
2022-09-27T15:49:28.0299634Z 
2022-09-27T15:49:28.0299744Z OK (skipped=1)
2022-09-27T15:49:28.0299899Z 
2022-09-27T15:49:28.0300023Z Generating XML reports...
2022-09-27T15:49:28.0335195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154928.xml
2022-09-27T15:49:28.5312962Z ok (6.657s)
2022-09-27T15:49:28.5313229Z 
2022-09-27T15:49:28.5313618Z ----------------------------------------------------------------------
2022-09-27T15:49:28.5313962Z Ran 1 test in 6.657s
2022-09-27T15:49:28.5314132Z 
2022-09-27T15:49:28.5314234Z OK
2022-09-27T15:49:28.5314370Z 
2022-09-27T15:49:28.5314488Z Generating XML reports...
2022-09-27T15:49:28.5349048Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154921.xml
2022-09-27T15:49:29.2928714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:29.2929259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:29.2930517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:29.2930982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:29.8502068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:49:29.8526266Z 
2022-09-27T15:49:29.8526546Z Running tests...
2022-09-27T15:49:29.8526973Z ----------------------------------------------------------------------
2022-09-27T15:49:30.1005541Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:30.1006840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:30.1008041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:30.1008998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:30.5950853Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:30.5972114Z 
2022-09-27T15:49:30.5972465Z Running tests...
2022-09-27T15:49:30.5973294Z ----------------------------------------------------------------------
2022-09-27T15:49:30.6408594Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:30.6409892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:30.6411120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:30.6412107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:31.1490967Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:49:31.1513374Z 
2022-09-27T15:49:31.1513815Z Running tests...
2022-09-27T15:49:31.1514317Z ----------------------------------------------------------------------
2022-09-27T15:49:31.1537657Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-09-27T15:49:31.1538077Z 
2022-09-27T15:49:31.1538345Z ----------------------------------------------------------------------
2022-09-27T15:49:31.1538679Z Ran 1 test in 0.002s
2022-09-27T15:49:31.1538844Z 
2022-09-27T15:49:31.1538956Z OK (skipped=1)
2022-09-27T15:49:31.1539132Z 
2022-09-27T15:49:31.1539259Z Generating XML reports...
2022-09-27T15:49:31.1574111Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154931.xml
2022-09-27T15:49:31.4621379Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:31.4828552Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4145
2022-09-27T15:49:31.4833743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4146
2022-09-27T15:49:32.1774530Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:32.2126625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4241
2022-09-27T15:49:32.2134958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4242
2022-09-27T15:49:33.1743618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:33.1744522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:33.1745619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:33.1746111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:33.1928455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:33.1928930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:33.1932541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:33.1933023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:33.1940406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:33.1940868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:33.1945441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:33.1945942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:33.6860975Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:49:33.6884522Z 
2022-09-27T15:49:33.6884669Z Running tests...
2022-09-27T15:49:33.6885304Z ----------------------------------------------------------------------
2022-09-27T15:49:33.7064522Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:33.7215131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:33.7215938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:33.7268180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:33.7275190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:33.8230295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:33.8781648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:33.8782141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:33.8783611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:33.8784075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:33.9053951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:33.9054448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:33.9056616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:33.9057092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:34.4672073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:34.4672610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:34.4673404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:34.4674084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:34.4681083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:34.4681613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:35.0904112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8s6_qjq
2022-09-27T15:49:35.0904711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8s6_qjq/_remote_module_non_scriptable.py
2022-09-27T15:49:35.2223509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpded0dc0u
2022-09-27T15:49:35.2224078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpded0dc0u/_remote_module_non_scriptable.py
2022-09-27T15:49:35.4238036Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:35.4448503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4333
2022-09-27T15:49:35.4455186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4334
2022-09-27T15:49:35.8393362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgu4fhux1
2022-09-27T15:49:35.8394135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgu4fhux1/_remote_module_non_scriptable.py
2022-09-27T15:49:35.8790488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xec5el5
2022-09-27T15:49:35.8791075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xec5el5/_remote_module_non_scriptable.py
2022-09-27T15:49:36.4447190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:36.4447775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:36.4674335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:36.4674877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:36.7886205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:36.7886981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:36.8057171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:36.8058026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:37.0256881Z ok (6.428s)
2022-09-27T15:49:37.0257113Z 
2022-09-27T15:49:37.0257504Z ----------------------------------------------------------------------
2022-09-27T15:49:37.0257849Z Ran 1 test in 6.428s
2022-09-27T15:49:37.0257997Z 
2022-09-27T15:49:37.0258092Z OK
2022-09-27T15:49:37.0258226Z 
2022-09-27T15:49:37.0258359Z Generating XML reports...
2022-09-27T15:49:37.0293013Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154930.xml
2022-09-27T15:49:37.1493060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:37.1493593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:37.1495539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:37.1496036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:37.1564852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:37.1565333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:37.1568803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:37.1569284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:37.3962871Z ok (7.543s)
2022-09-27T15:49:37.3963273Z 
2022-09-27T15:49:37.3963683Z ----------------------------------------------------------------------
2022-09-27T15:49:37.3964025Z Ran 1 test in 7.544s
2022-09-27T15:49:37.3964190Z 
2022-09-27T15:49:37.3964305Z OK
2022-09-27T15:49:37.3964443Z 
2022-09-27T15:49:37.3964560Z Generating XML reports...
2022-09-27T15:49:37.3998976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154929.xml
2022-09-27T15:49:37.6782039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:37.7003368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:37.7004205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:37.7085740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:37.7092738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:37.8017959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:39.1114637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0lrg7ki
2022-09-27T15:49:39.1115986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0lrg7ki/_remote_module_non_scriptable.py
2022-09-27T15:49:39.2333021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:39.2333552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:39.2336127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:39.2336603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:39.2724961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv_nf025
2022-09-27T15:49:39.2726305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv_nf025/_remote_module_non_scriptable.py
2022-09-27T15:49:39.3005497Z [1664293779.300087] [28fd6574525b:4333 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:39.3006331Z [1664293779.300087] [28fd6574525b:4333 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:39.3086356Z [1664293779.308240] [28fd6574525b:4334 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:39.3086859Z [1664293779.308240] [28fd6574525b:4334 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:39.5568019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:39.5568519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:39.5570152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:39.5570644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:39.7282816Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:39.7305359Z 
2022-09-27T15:49:39.7305681Z Running tests...
2022-09-27T15:49:39.7306135Z ----------------------------------------------------------------------
2022-09-27T15:49:40.0625960Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:49:40.0649276Z 
2022-09-27T15:49:40.0649558Z Running tests...
2022-09-27T15:49:40.0650009Z ----------------------------------------------------------------------
2022-09-27T15:49:40.0824304Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:40.0824985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:40.0992807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:40.0993413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:40.6574041Z ok (6.969s)
2022-09-27T15:49:40.6574393Z 
2022-09-27T15:49:40.6575147Z ----------------------------------------------------------------------
2022-09-27T15:49:40.6575718Z Ran 1 test in 6.969s
2022-09-27T15:49:40.6575889Z 
2022-09-27T15:49:40.6575966Z OK
2022-09-27T15:49:40.6576102Z 
2022-09-27T15:49:40.6576236Z Generating XML reports...
2022-09-27T15:49:40.6611228Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154933.xml
2022-09-27T15:49:41.4484448Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:41.4687859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4499
2022-09-27T15:49:41.4694634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4500
2022-09-27T15:49:41.7261367Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:41.7454240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4502
2022-09-27T15:49:41.7459770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4503
2022-09-27T15:49:42.8917023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:42.8918001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:42.8919326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:42.8919845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:43.1366718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:43.1367649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:43.1368929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:43.1369832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:43.1373242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:43.1373719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:43.1377125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:43.1377635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:43.3971729Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:49:43.3995381Z 
2022-09-27T15:49:43.3995837Z Running tests...
2022-09-27T15:49:43.3996525Z ----------------------------------------------------------------------
2022-09-27T15:49:43.5213467Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:43.5214817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:43.5215982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:43.5216941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:43.5303514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:43.5304365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:43.5307448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:43.5308730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:43.6833604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:43.6834724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:43.6836228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:43.6837649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:43.6841234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:43.6842217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:44.0485677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:44.0681591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:44.0682491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:44.0689741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:44.0697506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:44.1697696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:45.0853808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qeri3k6
2022-09-27T15:49:45.0855008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qeri3k6/_remote_module_non_scriptable.py
2022-09-27T15:49:45.0951063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_m_t6vq
2022-09-27T15:49:45.0952272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_m_t6vq/_remote_module_non_scriptable.py
2022-09-27T15:49:45.1175957Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:45.1228187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.1228749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.1387653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4688
2022-09-27T15:49:45.1393050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4689
2022-09-27T15:49:45.1393751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.1394250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:45.4341522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wrckr4q
2022-09-27T15:49:45.4342137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wrckr4q/_remote_module_non_scriptable.py
2022-09-27T15:49:45.5231555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcfssxza
2022-09-27T15:49:45.5232530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcfssxza/_remote_module_non_scriptable.py
2022-09-27T15:49:45.6802277Z ok (5.949s)
2022-09-27T15:49:45.6802501Z 
2022-09-27T15:49:45.6802894Z ----------------------------------------------------------------------
2022-09-27T15:49:45.6803235Z Ran 1 test in 5.950s
2022-09-27T15:49:45.6803401Z 
2022-09-27T15:49:45.6803479Z OK
2022-09-27T15:49:45.6803620Z 
2022-09-27T15:49:45.6803753Z Generating XML reports...
2022-09-27T15:49:45.6839665Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154939.xml
2022-09-27T15:49:46.7698547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:46.7699054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:46.7700352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:46.7700838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:46.7956029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:46.7956545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:46.8093004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:46.8096293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:46.8122708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:46.8123203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:46.8127330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:46.8127822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:47.3794970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:47.4023936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:47.4024750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:47.4098447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:47.4105652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:47.4604057Z ok (7.395s)
2022-09-27T15:49:47.4604264Z 
2022-09-27T15:49:47.4604625Z ----------------------------------------------------------------------
2022-09-27T15:49:47.4605233Z Ran 1 test in 7.395s
2022-09-27T15:49:47.4605382Z 
2022-09-27T15:49:47.4605485Z OK
2022-09-27T15:49:47.4605619Z 
2022-09-27T15:49:47.4605750Z Generating XML reports...
2022-09-27T15:49:47.4641792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154940.xml
2022-09-27T15:49:47.5040880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:47.8567478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:47.8567969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:47.8569789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:47.8570267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:48.3715209Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:48.3738316Z 
2022-09-27T15:49:48.3738620Z Running tests...
2022-09-27T15:49:48.3739056Z ----------------------------------------------------------------------
2022-09-27T15:49:48.8566898Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyad19p5g
2022-09-27T15:49:48.8567637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyad19p5g/_remote_module_non_scriptable.py
2022-09-27T15:49:48.9529692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uxko_91
2022-09-27T15:49:48.9531122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uxko_91/_remote_module_non_scriptable.py
2022-09-27T15:49:48.9786777Z [1664293788.978264] [28fd6574525b:4688 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:48.9787332Z [1664293788.978264] [28fd6574525b:4688 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:48.9844521Z [1664293788.984131] [28fd6574525b:4689 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:49:48.9845013Z [1664293788.984131] [28fd6574525b:4689 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:49:49.3251929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:49.3252455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:49.3388561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:49.3389128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:49:49.7022420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:49.7022958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:49.7024567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:49.7025033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:49.9506041Z ok (6.551s)
2022-09-27T15:49:49.9506249Z 
2022-09-27T15:49:49.9506670Z ----------------------------------------------------------------------
2022-09-27T15:49:49.9507012Z Ran 1 test in 6.551s
2022-09-27T15:49:49.9507182Z 
2022-09-27T15:49:49.9507260Z OK
2022-09-27T15:49:49.9507394Z 
2022-09-27T15:49:49.9507527Z Generating XML reports...
2022-09-27T15:49:49.9553033Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154943.xml
2022-09-27T15:49:50.0230348Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:50.0403168Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.666s)
2022-09-27T15:49:50.0403858Z 
2022-09-27T15:49:50.0404152Z ----------------------------------------------------------------------
2022-09-27T15:49:50.0404490Z Ran 1 test in 1.666s
2022-09-27T15:49:50.0404657Z 
2022-09-27T15:49:50.0404765Z OK (skipped=1)
2022-09-27T15:49:50.0404923Z 
2022-09-27T15:49:50.0405031Z Generating XML reports...
2022-09-27T15:49:50.0437489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154948.xml
2022-09-27T15:49:50.2355918Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:49:50.2378825Z 
2022-09-27T15:49:50.2379114Z Running tests...
2022-09-27T15:49:50.2379575Z ----------------------------------------------------------------------
2022-09-27T15:49:51.9789232Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:51.9955151Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.757s)
2022-09-27T15:49:51.9955740Z 
2022-09-27T15:49:51.9956029Z ----------------------------------------------------------------------
2022-09-27T15:49:51.9956361Z Ran 1 test in 1.757s
2022-09-27T15:49:51.9956525Z 
2022-09-27T15:49:51.9956649Z OK (skipped=1)
2022-09-27T15:49:51.9956806Z 
2022-09-27T15:49:51.9956914Z Generating XML reports...
2022-09-27T15:49:51.9989314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154950.xml
2022-09-27T15:49:52.1247866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:52.1248629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:52.1250703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:52.1251192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:52.1644175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:52.1644687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:52.1647519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:52.1648029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:52.6264437Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:49:52.6287827Z 
2022-09-27T15:49:52.6288243Z Running tests...
2022-09-27T15:49:52.6288714Z ----------------------------------------------------------------------
2022-09-27T15:49:52.6570377Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:52.6593247Z 
2022-09-27T15:49:52.6593460Z Running tests...
2022-09-27T15:49:52.6593891Z ----------------------------------------------------------------------
2022-09-27T15:49:54.1279710Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:54.1280660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:54.1281986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:54.1282463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:54.3052084Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:54.3101628Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:54.3215919Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.693s)
2022-09-27T15:49:54.3216477Z 
2022-09-27T15:49:54.3216762Z ----------------------------------------------------------------------
2022-09-27T15:49:54.3217097Z Ran 1 test in 1.693s
2022-09-27T15:49:54.3217261Z 
2022-09-27T15:49:54.3217369Z OK (skipped=1)
2022-09-27T15:49:54.3217509Z 
2022-09-27T15:49:54.3217643Z Generating XML reports...
2022-09-27T15:49:54.3251808Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154952.xml
2022-09-27T15:49:54.3313481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4953
2022-09-27T15:49:54.3320528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4954
2022-09-27T15:49:54.6480124Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:49:54.6503639Z 
2022-09-27T15:49:54.6504201Z Running tests...
2022-09-27T15:49:54.6504732Z ----------------------------------------------------------------------
2022-09-27T15:49:56.0097610Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:56.0098873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:56.0100076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:56.0100923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:56.0366175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:56.0367232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:56.0368933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:56.0369950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:56.3850145Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:56.4045929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5053
2022-09-27T15:49:56.4052057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5054
2022-09-27T15:49:56.4493669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:56.4494180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:56.4497571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:56.4498064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:56.5653526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:56.5654101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:56.5654876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:56.5655572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:56.5662280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:56.5662784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:56.9385052Z ok (4.279s)
2022-09-27T15:49:56.9385310Z 
2022-09-27T15:49:56.9385715Z ----------------------------------------------------------------------
2022-09-27T15:49:56.9386065Z Ran 1 test in 4.279s
2022-09-27T15:49:56.9386232Z 
2022-09-27T15:49:56.9386326Z OK
2022-09-27T15:49:56.9386461Z 
2022-09-27T15:49:56.9386578Z Generating XML reports...
2022-09-27T15:49:56.9423284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154952.xml
2022-09-27T15:49:56.9533059Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:49:56.9557372Z 
2022-09-27T15:49:56.9557672Z Running tests...
2022-09-27T15:49:56.9558108Z ----------------------------------------------------------------------
2022-09-27T15:49:58.0472173Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:58.0472819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:58.0473834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:58.0474317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:58.1051465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:58.1052210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:58.1055926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:58.1056577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:58.5709856Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:49:58.5902624Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5162
2022-09-27T15:49:58.5909360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5163
2022-09-27T15:49:58.6071144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:49:58.6683759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:49:58.6684600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:58.6782414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:49:58.6789475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:49:58.7696282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:49:59.0972661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:49:59.0973171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:49:59.0974645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:49:59.0975111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:49:59.2119251Z ok (4.561s)
2022-09-27T15:49:59.2119451Z 
2022-09-27T15:49:59.2119829Z ----------------------------------------------------------------------
2022-09-27T15:49:59.2120363Z Ran 1 test in 4.561s
2022-09-27T15:49:59.2120548Z 
2022-09-27T15:49:59.2120647Z OK
2022-09-27T15:49:59.2120784Z 
2022-09-27T15:49:59.2120901Z Generating XML reports...
2022-09-27T15:49:59.2155424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154954.xml
2022-09-27T15:49:59.5953973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:49:59.5976512Z 
2022-09-27T15:49:59.5976829Z Running tests...
2022-09-27T15:49:59.5977580Z ----------------------------------------------------------------------
2022-09-27T15:50:00.2309384Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:00.2310018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:00.2311022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:00.2311488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:00.2576424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:00.2576921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:00.2579061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:00.2579529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:00.7387761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:00.7938127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:00.7939536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:00.7994698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:00.8002444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:00.8952466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:01.2035265Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:01.2235400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5271
2022-09-27T15:50:01.2240505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5272
2022-09-27T15:50:01.3979067Z ok (4.442s)
2022-09-27T15:50:01.3979292Z 
2022-09-27T15:50:01.3979702Z ----------------------------------------------------------------------
2022-09-27T15:50:01.3980047Z Ran 1 test in 4.442s
2022-09-27T15:50:01.3980211Z 
2022-09-27T15:50:01.3980539Z OK
2022-09-27T15:50:01.3980701Z 
2022-09-27T15:50:01.3980846Z Generating XML reports...
2022-09-27T15:50:01.4016311Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154956.xml
2022-09-27T15:50:01.4349042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:01.4349526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:01.4351695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:01.4352182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:02.0088668Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:50:02.0124098Z 
2022-09-27T15:50:02.0124584Z Running tests...
2022-09-27T15:50:02.0125100Z ----------------------------------------------------------------------
2022-09-27T15:50:02.8111671Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:02.8113391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:02.8114548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:02.8115474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:02.8797466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:02.8798478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:02.8799640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:02.8801000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:03.4131947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:03.4133069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:03.4134552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:03.4136027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:03.4139679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:03.4140669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:03.6084519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:03.6085553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:03.6086743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:03.6087725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:03.6258543Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:03.6465117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5383
2022-09-27T15:50:03.6470235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5384
2022-09-27T15:50:04.1433728Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:04.1457724Z 
2022-09-27T15:50:04.1457989Z Running tests...
2022-09-27T15:50:04.1458428Z ----------------------------------------------------------------------
2022-09-27T15:50:04.8422818Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80eqy616
2022-09-27T15:50:04.8423573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80eqy616/_remote_module_non_scriptable.py
2022-09-27T15:50:04.8546541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3b8r72o3
2022-09-27T15:50:04.8548414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3b8r72o3/_remote_module_non_scriptable.py
2022-09-27T15:50:05.3175314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:05.3175822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:05.3177771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:05.3178269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:05.3199786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:05.3200546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:05.3205048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:05.3205546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:05.3208884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:05.3209386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:05.3851969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:05.3852532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:05.8389237Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:05.8543277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:05.8548553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:05.8549651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:05.8592132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5459
2022-09-27T15:50:05.8598739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5460
2022-09-27T15:50:05.8645959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:05.8652670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:05.9351478Z ok (6.337s)
2022-09-27T15:50:05.9351698Z 
2022-09-27T15:50:05.9352060Z ----------------------------------------------------------------------
2022-09-27T15:50:05.9352419Z Ran 1 test in 6.337s
2022-09-27T15:50:05.9352594Z 
2022-09-27T15:50:05.9352693Z OK
2022-09-27T15:50:05.9352812Z 
2022-09-27T15:50:05.9352946Z Generating XML reports...
2022-09-27T15:50:05.9388969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154959.xml
2022-09-27T15:50:05.9565892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:07.4509802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel5imqz2
2022-09-27T15:50:07.4510422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel5imqz2/_remote_module_non_scriptable.py
2022-09-27T15:50:07.4769179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplktp92zs
2022-09-27T15:50:07.4770270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplktp92zs/_remote_module_non_scriptable.py
2022-09-27T15:50:07.5601371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:07.5602370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:07.5603524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:07.5604457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:07.6133473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:07.6133961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:07.6135374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:07.6135857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:08.0001609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:08.0002172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:08.0549068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:08.0549955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:08.0795843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:08.1309475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:08.1310291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:08.1406688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:08.1413250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:08.2322011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:08.3162131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:08.3162896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:08.3164811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:08.3165293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:08.6580155Z ok (6.645s)
2022-09-27T15:50:08.6580372Z 
2022-09-27T15:50:08.6580732Z ----------------------------------------------------------------------
2022-09-27T15:50:08.6581079Z Ran 1 test in 6.645s
2022-09-27T15:50:08.6581249Z 
2022-09-27T15:50:08.6581345Z OK
2022-09-27T15:50:08.6581484Z 
2022-09-27T15:50:08.6581618Z Generating XML reports...
2022-09-27T15:50:08.6617788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155002.xml
2022-09-27T15:50:08.8524932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:08.8549544Z 
2022-09-27T15:50:08.8549929Z Running tests...
2022-09-27T15:50:08.8550410Z ----------------------------------------------------------------------
2022-09-27T15:50:09.6003902Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwab3y7b
2022-09-27T15:50:09.6004623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwab3y7b/_remote_module_non_scriptable.py
2022-09-27T15:50:09.6632257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsoitso_3
2022-09-27T15:50:09.6633115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsoitso_3/_remote_module_non_scriptable.py
2022-09-27T15:50:09.6983961Z [1664293809.697968] [28fd6574525b:5459 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:50:09.6984488Z [1664293809.697968] [28fd6574525b:5459 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:50:09.6989707Z [1664293809.698691] [28fd6574525b:5460 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:50:09.6990179Z [1664293809.698691] [28fd6574525b:5460 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:50:10.5602330Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:10.5796701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:10.5797231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:10.5811430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5624
2022-09-27T15:50:10.5817988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5625
2022-09-27T15:50:10.6341624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:10.6342361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:10.8808267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:10.8808871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:10.8811436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:10.8811929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:11.2749063Z ok (7.128s)
2022-09-27T15:50:11.2751538Z 
2022-09-27T15:50:11.2754222Z ----------------------------------------------------------------------
2022-09-27T15:50:11.2756838Z Ran 1 test in 7.128s
2022-09-27T15:50:11.2759046Z 
2022-09-27T15:50:11.2761605Z OK
2022-09-27T15:50:11.2763801Z 
2022-09-27T15:50:11.2765925Z Generating XML reports...
2022-09-27T15:50:11.2809729Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155004.xml
2022-09-27T15:50:11.4504099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:50:11.4527635Z 
2022-09-27T15:50:11.4528088Z Running tests...
2022-09-27T15:50:11.4528581Z ----------------------------------------------------------------------
2022-09-27T15:50:12.2370191Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:12.2370809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:12.2372042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:12.2372507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:12.2395487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:12.2395950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:12.2399886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:12.2400578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:12.8360880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:12.8361450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:12.8362456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:12.8363171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:12.8369626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:12.8370118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:13.2074586Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:13.2282684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5735
2022-09-27T15:50:13.2288212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5736
2022-09-27T15:50:13.5033546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:13.5034579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:13.5035783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:13.5036749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:14.0864728Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:14.0888905Z 
2022-09-27T15:50:14.0889165Z Running tests...
2022-09-27T15:50:14.0889631Z ----------------------------------------------------------------------
2022-09-27T15:50:14.7914149Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... ok (5.936s)
2022-09-27T15:50:14.7914468Z 
2022-09-27T15:50:14.7914863Z ----------------------------------------------------------------------
2022-09-27T15:50:14.7915202Z Ran 1 test in 5.936s
2022-09-27T15:50:14.7915368Z 
2022-09-27T15:50:14.7915464Z OK
2022-09-27T15:50:14.7915596Z 
2022-09-27T15:50:14.7915705Z Generating XML reports...
2022-09-27T15:50:14.7951387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155008.xml
2022-09-27T15:50:14.8823258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:14.8823772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:14.8826283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:14.8826781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:14.8862069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:14.8862531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:14.8865988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:14.8866464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:15.4040459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:15.4092168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:15.4093003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:15.4140936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:15.4147186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:15.5107164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:15.7970527Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:15.8182473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5813
2022-09-27T15:50:15.8188467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5814
2022-09-27T15:50:17.0717166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:17.0717721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:17.0718756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:17.0719223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:17.5473875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:17.5474395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:17.5475358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:17.5475870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:17.5567610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:17.5568406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:17.5571999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:17.5572492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:17.5955324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:17.5979450Z 
2022-09-27T15:50:17.5979866Z Running tests...
2022-09-27T15:50:17.5980396Z ----------------------------------------------------------------------
2022-09-27T15:50:17.5998714Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-09-27T15:50:17.5999476Z 
2022-09-27T15:50:17.5999926Z ----------------------------------------------------------------------
2022-09-27T15:50:17.6000560Z Ran 1 test in 0.002s
2022-09-27T15:50:17.6000737Z 
2022-09-27T15:50:17.6000853Z OK (skipped=1)
2022-09-27T15:50:17.6001019Z 
2022-09-27T15:50:17.6001146Z Generating XML reports...
2022-09-27T15:50:17.6035736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155017.xml
2022-09-27T15:50:18.0748706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:18.0814084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:18.0814902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:18.0849874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:18.0856695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:18.1829262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:18.5408045Z ok (7.088s)
2022-09-27T15:50:18.5408267Z 
2022-09-27T15:50:18.5408662Z ----------------------------------------------------------------------
2022-09-27T15:50:18.5409020Z Ran 1 test in 7.088s
2022-09-27T15:50:18.5409189Z 
2022-09-27T15:50:18.5409287Z OK
2022-09-27T15:50:18.5409405Z 
2022-09-27T15:50:18.5409546Z Generating XML reports...
2022-09-27T15:50:18.5444328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155011.xml
2022-09-27T15:50:19.5675874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:19.5676422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:19.5679435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:19.5679945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:19.6425205Z [1664293819.642001] [28fd6574525b:5813 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:50:19.6425727Z [1664293819.642001] [28fd6574525b:5813 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:50:19.6441802Z [1664293819.643865] [28fd6574525b:5814 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:50:19.6442267Z [1664293819.643865] [28fd6574525b:5814 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:50:20.1417108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:20.1440250Z 
2022-09-27T15:50:20.1440768Z Running tests...
2022-09-27T15:50:20.1441280Z ----------------------------------------------------------------------
2022-09-27T15:50:20.1459477Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:20.5300716Z Runs _test_accumulate_gradients_no_sync using default inputs ... ok (6.441s)
2022-09-27T15:50:20.5301279Z 
2022-09-27T15:50:20.5301952Z ----------------------------------------------------------------------
2022-09-27T15:50:20.5302565Z Ran 1 test in 6.441s
2022-09-27T15:50:20.5302873Z 
2022-09-27T15:50:20.5303049Z OK
2022-09-27T15:50:20.5303299Z 
2022-09-27T15:50:20.5303536Z Generating XML reports...
2022-09-27T15:50:20.5339124Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155014.xml
2022-09-27T15:50:20.8158950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:20.8159474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:20.8161465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:20.8161959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:21.3129498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:50:21.3151875Z 
2022-09-27T15:50:21.3152304Z Running tests...
2022-09-27T15:50:21.3152789Z ----------------------------------------------------------------------
2022-09-27T15:50:21.3172376Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-09-27T15:50:21.3172704Z 
2022-09-27T15:50:21.3172994Z ----------------------------------------------------------------------
2022-09-27T15:50:21.3173327Z Ran 1 test in 0.002s
2022-09-27T15:50:21.3173494Z 
2022-09-27T15:50:21.3173606Z OK (skipped=1)
2022-09-27T15:50:21.3173762Z 
2022-09-27T15:50:21.3173890Z Generating XML reports...
2022-09-27T15:50:21.3208574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155021.xml
2022-09-27T15:50:21.9080138Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:21.9278259Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6033
2022-09-27T15:50:21.9284457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6034
2022-09-27T15:50:22.6540000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:22.6549099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:22.6550613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:22.6551772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:23.2086999Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:23.2108261Z 
2022-09-27T15:50:23.2108536Z Running tests...
2022-09-27T15:50:23.2108993Z ----------------------------------------------------------------------
2022-09-27T15:50:23.2127698Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-09-27T15:50:23.2128158Z 
2022-09-27T15:50:23.2128454Z ----------------------------------------------------------------------
2022-09-27T15:50:23.2128786Z Ran 1 test in 0.002s
2022-09-27T15:50:23.2128957Z 
2022-09-27T15:50:23.2129071Z OK (skipped=1)
2022-09-27T15:50:23.2129231Z 
2022-09-27T15:50:23.2129341Z Generating XML reports...
2022-09-27T15:50:23.2162774Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155023.xml
2022-09-27T15:50:23.3594085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:23.3594613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:23.3597695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:23.3598489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:23.6648530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:23.6649040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:23.6651885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:23.6652378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:23.6838695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:23.6839180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:23.6842991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:23.6843475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:23.8674474Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:50:23.8704129Z 
2022-09-27T15:50:23.8704683Z Running tests...
2022-09-27T15:50:23.8705129Z ----------------------------------------------------------------------
2022-09-27T15:50:23.8726207Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:24.2575467Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:24.2576737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:24.2577308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:24.2577980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:24.2683226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:24.2794445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedoybq63
2022-09-27T15:50:24.2797153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedoybq63/_remote_module_non_scriptable.py
2022-09-27T15:50:24.3593947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:24.3704197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_n_4gp4s
2022-09-27T15:50:24.3706864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_n_4gp4s/_remote_module_non_scriptable.py
2022-09-27T15:50:24.8361610Z ok (4.692s)
2022-09-27T15:50:24.8361850Z 
2022-09-27T15:50:24.8362236Z ----------------------------------------------------------------------
2022-09-27T15:50:24.8362557Z Ran 1 test in 4.692s
2022-09-27T15:50:24.8362733Z 
2022-09-27T15:50:24.8362829Z OK
2022-09-27T15:50:24.8362964Z 
2022-09-27T15:50:24.8363100Z Generating XML reports...
2022-09-27T15:50:24.8413671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155020.xml
2022-09-27T15:50:25.2418131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:25.2418637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:25.2420660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:25.2421151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:25.5224973Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:25.5423791Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6180
2022-09-27T15:50:25.5430272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6181
2022-09-27T15:50:25.7543179Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:25.7565513Z 
2022-09-27T15:50:25.7565779Z Running tests...
2022-09-27T15:50:25.7566202Z ----------------------------------------------------------------------
2022-09-27T15:50:25.7583521Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:25.7584786Z Runs _test_accumulate_gradients_no_sync using default inputs ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T15:50:25.7585420Z 
2022-09-27T15:50:25.7585995Z ----------------------------------------------------------------------
2022-09-27T15:50:25.7586641Z Ran 1 test in 0.002s
2022-09-27T15:50:25.7586935Z 
2022-09-27T15:50:25.7587128Z OK (skipped=1)
2022-09-27T15:50:25.7587431Z 
2022-09-27T15:50:25.7587666Z Generating XML reports...
2022-09-27T15:50:25.7620910Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155025.xml
2022-09-27T15:50:27.1988387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:27.1989404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:27.1991135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:27.1992068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:27.3230984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:27.3231996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:27.3233248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:27.3234200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:27.4304355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:27.4305360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:27.4306589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:27.4307559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:27.6928988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:27.6953661Z 
2022-09-27T15:50:27.6954153Z Running tests...
2022-09-27T15:50:27.6954651Z ----------------------------------------------------------------------
2022-09-27T15:50:27.6976100Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:27.7820156Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:27.7820740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:27.7822775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:27.7823249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:27.8252403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:27.9434485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:27.9435318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:27.9468770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:27.9475986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:28.0449205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:28.2916113Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:28.2939006Z 
2022-09-27T15:50:28.2939538Z Running tests...
2022-09-27T15:50:28.2940022Z ----------------------------------------------------------------------
2022-09-27T15:50:28.2959915Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:28.2961321Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T15:50:28.2962033Z 
2022-09-27T15:50:28.2962597Z ----------------------------------------------------------------------
2022-09-27T15:50:28.2963236Z Ran 1 test in 0.002s
2022-09-27T15:50:28.2963591Z 
2022-09-27T15:50:28.2963821Z OK (skipped=1)
2022-09-27T15:50:28.2964151Z 
2022-09-27T15:50:28.2964382Z Generating XML reports...
2022-09-27T15:50:28.2998797Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155028.xml
2022-09-27T15:50:29.2575478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xb7s7xg
2022-09-27T15:50:29.2576504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xb7s7xg/_remote_module_non_scriptable.py
2022-09-27T15:50:29.3430463Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:29.3725409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6354
2022-09-27T15:50:29.3816258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6355
2022-09-27T15:50:29.4474230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10kstupi
2022-09-27T15:50:29.4475699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10kstupi/_remote_module_non_scriptable.py
2022-09-27T15:50:30.3051048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:30.3051681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:30.3053193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:30.3053703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:30.8141072Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:30.8164306Z 
2022-09-27T15:50:30.8164826Z Running tests...
2022-09-27T15:50:30.8165638Z ----------------------------------------------------------------------
2022-09-27T15:50:30.8187713Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:30.8188569Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T15:50:30.8189009Z 
2022-09-27T15:50:30.8189332Z ----------------------------------------------------------------------
2022-09-27T15:50:30.8189668Z Ran 1 test in 0.002s
2022-09-27T15:50:30.8189831Z 
2022-09-27T15:50:30.8189943Z OK (skipped=1)
2022-09-27T15:50:30.8190100Z 
2022-09-27T15:50:30.8190225Z Generating XML reports...
2022-09-27T15:50:30.8225089Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155030.xml
2022-09-27T15:50:31.1202277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:31.1202795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:31.1204545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:31.1205389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:31.1219817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:31.1220281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:31.1223811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:31.1224292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:31.4553708Z ok (7.585s)
2022-09-27T15:50:31.4553931Z 
2022-09-27T15:50:31.4554518Z ----------------------------------------------------------------------
2022-09-27T15:50:31.4554868Z Ran 1 test in 7.585s
2022-09-27T15:50:31.4555038Z 
2022-09-27T15:50:31.4555141Z OK
2022-09-27T15:50:31.4555281Z 
2022-09-27T15:50:31.4555414Z Generating XML reports...
2022-09-27T15:50:31.4590180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155023.xml
2022-09-27T15:50:31.6984868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:31.6985405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:31.6986182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:31.6986857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:31.7093054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:31.7204869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_zftee3
2022-09-27T15:50:31.7208232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_zftee3/_remote_module_non_scriptable.py
2022-09-27T15:50:31.8008212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:31.8210177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mwdjqsp
2022-09-27T15:50:31.8216959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mwdjqsp/_remote_module_non_scriptable.py
2022-09-27T15:50:31.8863255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:31.8867176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:32.2890438Z ok (4.593s)
2022-09-27T15:50:32.2890651Z 
2022-09-27T15:50:32.2891054Z ----------------------------------------------------------------------
2022-09-27T15:50:32.2891398Z Ran 1 test in 4.594s
2022-09-27T15:50:32.2891569Z 
2022-09-27T15:50:32.2891916Z OK
2022-09-27T15:50:32.2892073Z 
2022-09-27T15:50:32.2892199Z Generating XML reports...
2022-09-27T15:50:32.2927448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155027.xml
2022-09-27T15:50:32.8283207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:32.8283695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:32.8287279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:32.8287768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:33.3224110Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:33.3247859Z 
2022-09-27T15:50:33.3248424Z Running tests...
2022-09-27T15:50:33.3249128Z ----------------------------------------------------------------------
2022-09-27T15:50:33.3266611Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:33.3267316Z Runs _test_accumulate_gradients_no_sync using default inputs ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T15:50:33.3267900Z 
2022-09-27T15:50:33.3268206Z ----------------------------------------------------------------------
2022-09-27T15:50:33.3268542Z Ran 1 test in 0.002s
2022-09-27T15:50:33.3268689Z 
2022-09-27T15:50:33.3268799Z OK (skipped=1)
2022-09-27T15:50:33.3268953Z 
2022-09-27T15:50:33.3269078Z Generating XML reports...
2022-09-27T15:50:33.3304811Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155033.xml
2022-09-27T15:50:33.5858132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:33.5858630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:33.5859913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:33.5860399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:34.0783264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:50:34.0805903Z 
2022-09-27T15:50:34.0806154Z Running tests...
2022-09-27T15:50:34.0806577Z ----------------------------------------------------------------------
2022-09-27T15:50:34.0826974Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:34.3384567Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:34.3385903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:34.3387320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:34.3388402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:34.8377552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:34.8400472Z 
2022-09-27T15:50:34.8400986Z Running tests...
2022-09-27T15:50:34.8401556Z ----------------------------------------------------------------------
2022-09-27T15:50:34.8424321Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:35.2470849Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:35.2471456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:35.2475037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:35.2475563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:35.7243821Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:35.7621697Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6579
2022-09-27T15:50:35.7622565Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:35.7627434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6580
2022-09-27T15:50:35.7650237Z 
2022-09-27T15:50:35.7650508Z Running tests...
2022-09-27T15:50:35.7650943Z ----------------------------------------------------------------------
2022-09-27T15:50:36.4599574Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:36.5055457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6645
2022-09-27T15:50:36.5132721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6646
2022-09-27T15:50:37.4253611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:37.4254448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:37.4255589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:37.4256070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:37.4406565Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:37.4601070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6711
2022-09-27T15:50:37.4607464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6712
2022-09-27T15:50:37.5302354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:37.5302883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:37.5305223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:37.5305735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:37.9291408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:38.0479111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:38.0479975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:38.0506416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:38.0513241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:38.1575813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:38.2053720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:38.2054936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:38.2056307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:38.2057397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:38.2058815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:38.2063043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:38.2064619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:38.2066308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:38.8001815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:38.8002425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:38.8003237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:38.8003950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:38.8111797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:38.8225726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tti1jw_
2022-09-27T15:50:38.8226697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tti1jw_/_remote_module_non_scriptable.py
2022-09-27T15:50:38.9017414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:38.9133102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmii9ebva
2022-09-27T15:50:38.9134518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmii9ebva/_remote_module_non_scriptable.py
2022-09-27T15:50:38.9419339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:38.9420408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:39.1118320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:39.1119156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:39.1120565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:39.1121421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:39.1455486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:39.1456421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:39.1457562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:39.1458505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:39.4204094Z ok (4.580s)
2022-09-27T15:50:39.4204312Z 
2022-09-27T15:50:39.4204732Z ----------------------------------------------------------------------
2022-09-27T15:50:39.4205063Z Ran 1 test in 4.580s
2022-09-27T15:50:39.4205229Z 
2022-09-27T15:50:39.4205325Z OK
2022-09-27T15:50:39.4205461Z 
2022-09-27T15:50:39.4205606Z Generating XML reports...
2022-09-27T15:50:39.4241426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155034.xml
2022-09-27T15:50:39.4249502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6tut762
2022-09-27T15:50:39.4252635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6tut762/_remote_module_non_scriptable.py
2022-09-27T15:50:39.4952851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshumsbk4
2022-09-27T15:50:39.4953469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshumsbk4/_remote_module_non_scriptable.py
2022-09-27T15:50:39.6504581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:39.6905199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:39.6906027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:39.6910660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:39.6918923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:39.6986947Z STAGE:2022-09-27 15:50:39 6711:6711 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:39.7918895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:39.7984838Z STAGE:2022-09-27 15:50:39 6712:6712 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:41.1744094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:41.1745144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:41.5392599Z [1664293841.538676] [28fd6574525b:6711 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:50:41.5393604Z [1664293841.538676] [28fd6574525b:6711 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:50:41.5919011Z [1664293841.591453] [28fd6574525b:6712 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:50:41.5919851Z [1664293841.591453] [28fd6574525b:6712 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:50:41.6267275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:41.6267754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:41.6271128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:41.6271628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:41.8758248Z ok (7.795s)
2022-09-27T15:50:41.8758434Z 
2022-09-27T15:50:41.8758812Z ----------------------------------------------------------------------
2022-09-27T15:50:41.8759153Z Ran 1 test in 7.795s
2022-09-27T15:50:41.8759322Z 
2022-09-27T15:50:41.8759422Z OK
2022-09-27T15:50:41.8759557Z 
2022-09-27T15:50:41.8759684Z Generating XML reports...
2022-09-27T15:50:41.8794532Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155034.xml
2022-09-27T15:50:42.0861136Z STAGE:2022-09-27 15:50:42 6711:6711 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:50:42 6712:6712 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:42.0861565Z 
2022-09-27T15:50:42.1591586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:42.1614927Z 
2022-09-27T15:50:42.1615504Z Running tests...
2022-09-27T15:50:42.1616023Z ----------------------------------------------------------------------
2022-09-27T15:50:42.1633844Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:42.2132266Z Runs _test_accumulate_gradients_no_sync using default inputs ... STAGE:2022-09-27 15:50:42 6711:6711 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:42.2136858Z STAGE:2022-09-27 15:50:42 6712:6712 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:42.2147631Z STAGE:2022-09-27 15:50:42 6711:6711 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:42.2148230Z STAGE:2022-09-27 15:50:42 6712:6712 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:43.3735446Z ok (7.608s)
2022-09-27T15:50:43.3735656Z 
2022-09-27T15:50:43.3736060Z ----------------------------------------------------------------------
2022-09-27T15:50:43.3736411Z Ran 1 test in 7.608s
2022-09-27T15:50:43.3736581Z 
2022-09-27T15:50:43.3736673Z OK
2022-09-27T15:50:43.3736814Z 
2022-09-27T15:50:43.3736932Z Generating XML reports...
2022-09-27T15:50:43.3772460Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155035.xml
2022-09-27T15:50:43.8776173Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:43.8970524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6895
2022-09-27T15:50:43.8976084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6896
2022-09-27T15:50:44.0135255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:44.0135758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:44.0138571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:44.0139068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:44.5295516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:50:44.5336889Z 
2022-09-27T15:50:44.5338823Z Running tests...
2022-09-27T15:50:44.5485073Z ----------------------------------------------------------------------
2022-09-27T15:50:44.5526524Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:45.4994747Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:45.4995333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:45.4996316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:45.4996803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:45.5881442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:45.5882106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:45.5882784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:45.5883257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:45.5883854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:45.5884324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:45.5884918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:45.5885392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:45.9921350Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:45.9943423Z 
2022-09-27T15:50:45.9943859Z Running tests...
2022-09-27T15:50:45.9944349Z ----------------------------------------------------------------------
2022-09-27T15:50:45.9961913Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-09-27T15:50:45.9962832Z 
2022-09-27T15:50:45.9963156Z ----------------------------------------------------------------------
2022-09-27T15:50:45.9963518Z Ran 1 test in 0.002s
2022-09-27T15:50:45.9963690Z 
2022-09-27T15:50:45.9963783Z OK (skipped=1)
2022-09-27T15:50:45.9963944Z 
2022-09-27T15:50:45.9964077Z Generating XML reports...
2022-09-27T15:50:45.9997234Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155045.xml
2022-09-27T15:50:46.1264155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:46.1267815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:46.1268821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:46.1366866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:46.1374124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:46.1488187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsx4josdi
2022-09-27T15:50:46.1490529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsx4josdi/_remote_module_non_scriptable.py
2022-09-27T15:50:46.2115682Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:46.2280972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:46.2315133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7004
2022-09-27T15:50:46.2322337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7005
2022-09-27T15:50:46.2393439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgogbmw60
2022-09-27T15:50:46.2395859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgogbmw60/_remote_module_non_scriptable.py
2022-09-27T15:50:46.6042183Z ok (4.442s)
2022-09-27T15:50:46.6042429Z 
2022-09-27T15:50:46.6042823Z ----------------------------------------------------------------------
2022-09-27T15:50:46.6043174Z Ran 1 test in 4.443s
2022-09-27T15:50:46.6043347Z 
2022-09-27T15:50:46.6043424Z OK
2022-09-27T15:50:46.6043562Z 
2022-09-27T15:50:46.6043697Z Generating XML reports...
2022-09-27T15:50:46.6078958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155042.xml
2022-09-27T15:50:47.9304761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:47.9306723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:47.9307614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:47.9308122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:47.9711800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:47.9712288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:47.9715285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:47.9715778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:48.0696789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:48.0697427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:48.0699792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:48.0700325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:48.4364289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:48.4749353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:48.4750195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:48.4769813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:48.4776953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:48.5694110Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:48.5718043Z 
2022-09-27T15:50:48.5718186Z Running tests...
2022-09-27T15:50:48.5718653Z ----------------------------------------------------------------------
2022-09-27T15:50:48.5738370Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-09-27T15:50:48.5739140Z 
2022-09-27T15:50:48.5739532Z ----------------------------------------------------------------------
2022-09-27T15:50:48.5739870Z Ran 1 test in 0.002s
2022-09-27T15:50:48.5740040Z 
2022-09-27T15:50:48.5740161Z OK (skipped=1)
2022-09-27T15:50:48.5740443Z 
2022-09-27T15:50:48.5740672Z Generating XML reports...
2022-09-27T15:50:48.5761628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:48.5777146Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155048.xml
2022-09-27T15:50:48.7851300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:48.7851954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:48.7853579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:48.7854133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:49.3120106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:49.3144318Z 
2022-09-27T15:50:49.3144747Z Running tests...
2022-09-27T15:50:49.3145266Z ----------------------------------------------------------------------
2022-09-27T15:50:49.8488518Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8i0egvo
2022-09-27T15:50:49.8489196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8i0egvo/_remote_module_non_scriptable.py
2022-09-27T15:50:49.9881404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg327lxh_
2022-09-27T15:50:49.9882032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg327lxh_/_remote_module_non_scriptable.py
2022-09-27T15:50:50.5057771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:50.5058289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:50.5059311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:50.5059791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:50.9847119Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:51.0047961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7186
2022-09-27T15:50:51.0053186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7187
2022-09-27T15:50:51.0343335Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:51.0366825Z 
2022-09-27T15:50:51.0366968Z Running tests...
2022-09-27T15:50:51.0367663Z ----------------------------------------------------------------------
2022-09-27T15:50:51.0386538Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-09-27T15:50:51.0386888Z 
2022-09-27T15:50:51.0387173Z ----------------------------------------------------------------------
2022-09-27T15:50:51.0387518Z Ran 1 test in 0.002s
2022-09-27T15:50:51.0387688Z 
2022-09-27T15:50:51.0387780Z OK (skipped=1)
2022-09-27T15:50:51.0387940Z 
2022-09-27T15:50:51.0388069Z Generating XML reports...
2022-09-27T15:50:51.0423035Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155051.xml
2022-09-27T15:50:51.5741700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:51.5750358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:50:52.2454902Z ok (7.697s)
2022-09-27T15:50:52.2455146Z 
2022-09-27T15:50:52.2455538Z ----------------------------------------------------------------------
2022-09-27T15:50:52.2455889Z Ran 1 test in 7.697s
2022-09-27T15:50:52.2456063Z 
2022-09-27T15:50:52.2456160Z OK
2022-09-27T15:50:52.2456299Z 
2022-09-27T15:50:52.2456436Z Generating XML reports...
2022-09-27T15:50:52.2493061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155044.xml
2022-09-27T15:50:52.7202754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:52.7203575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:52.7205834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:52.7206341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:52.7454250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:52.7455013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:52.7458039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:52.7458537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:53.1548805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:53.1549577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:53.1550893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:53.1551397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:53.2599412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:53.2599965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:53.2601026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:53.2601735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:53.2707285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:53.2774772Z STAGE:2022-09-27 15:50:53 7187:7187 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:53.3657734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:53.3786730Z STAGE:2022-09-27 15:50:53 7186:7186 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:53.3815567Z STAGE:2022-09-27 15:50:53 7186:7186 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:53.3816966Z STAGE:2022-09-27 15:50:53 7187:7187 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:53.3868165Z STAGE:2022-09-27 15:50:53 7187:7187 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:50:53 7186:7186 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:50:53.3895662Z 
2022-09-27T15:50:53.3925224Z STAGE:2022-09-27 15:50:53 7187:7187 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:50:53 7186:7186 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:50:53.3934352Z 
2022-09-27T15:50:53.7204594Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:53.7229592Z 
2022-09-27T15:50:53.7230003Z Running tests...
2022-09-27T15:50:53.7230512Z ----------------------------------------------------------------------
2022-09-27T15:50:53.7249552Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-09-27T15:50:53.7249980Z 
2022-09-27T15:50:53.7250489Z ----------------------------------------------------------------------
2022-09-27T15:50:53.7250839Z Ran 1 test in 0.002s
2022-09-27T15:50:53.7250989Z 
2022-09-27T15:50:53.7251110Z OK (skipped=1)
2022-09-27T15:50:53.7251269Z 
2022-09-27T15:50:53.7251399Z Generating XML reports...
2022-09-27T15:50:53.7288941Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155053.xml
2022-09-27T15:50:53.9122869Z ok (4.597s)
2022-09-27T15:50:53.9123098Z 
2022-09-27T15:50:53.9123501Z ----------------------------------------------------------------------
2022-09-27T15:50:53.9123862Z Ran 1 test in 4.598s
2022-09-27T15:50:53.9124032Z 
2022-09-27T15:50:53.9124131Z OK
2022-09-27T15:50:53.9124270Z 
2022-09-27T15:50:53.9124386Z Generating XML reports...
2022-09-27T15:50:53.9159390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155049.xml
2022-09-27T15:50:54.3502737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:54.3503250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:54.3505868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:54.3506414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:54.8341496Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:50:54.8363242Z 
2022-09-27T15:50:54.8363404Z Running tests...
2022-09-27T15:50:54.8364271Z ----------------------------------------------------------------------
2022-09-27T15:50:54.8381184Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T15:50:55.7204652Z Runs _test_accumulate_gradients_no_sync using default inputs ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:55.7205987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:55.7206615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:55.7207104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:55.9314251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:55.9314766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:55.9315861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:55.9316358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:56.2265652Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:56.2289427Z 
2022-09-27T15:50:56.2289905Z Running tests...
2022-09-27T15:50:56.2290611Z ----------------------------------------------------------------------
2022-09-27T15:50:56.2316340Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.003s)
2022-09-27T15:50:56.2316901Z 
2022-09-27T15:50:56.2317182Z ----------------------------------------------------------------------
2022-09-27T15:50:56.2317545Z Ran 1 test in 0.003s
2022-09-27T15:50:56.2317717Z 
2022-09-27T15:50:56.2317833Z OK (skipped=1)
2022-09-27T15:50:56.2317996Z 
2022-09-27T15:50:56.2318127Z Generating XML reports...
2022-09-27T15:50:56.2352673Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155056.xml
2022-09-27T15:50:56.4011300Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:56.4208065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7407
2022-09-27T15:50:56.4213743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7408
2022-09-27T15:50:56.4355116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:50:56.4378742Z 
2022-09-27T15:50:56.4379152Z Running tests...
2022-09-27T15:50:58.0816785Z ----------------------------------------------------------------------
2022-09-27T15:50:58.0817392Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:50:58.1013272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7505
2022-09-27T15:50:58.1019763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7506
2022-09-27T15:50:58.1036893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:58.1038173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:58.1040917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:58.1041898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:58.1224373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:58.1225316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:58.1228710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:58.1229723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:58.2541208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:58.2542142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:58.2543605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:58.2544541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:58.6203946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:50:58.6263310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:50:58.6264816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:58.6305856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:50:58.6312621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:50:58.7611989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:50:58.7843999Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:50:58.7887718Z 
2022-09-27T15:50:58.7888632Z Running tests...
2022-09-27T15:50:58.7890294Z ----------------------------------------------------------------------
2022-09-27T15:50:59.7525962Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:59.7526542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:59.7527685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:59.7528209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:50:59.7744847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:50:59.7745362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:50:59.7747523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:50:59.7748020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:00.0711107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpve2ubd4m
2022-09-27T15:51:00.0711717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpve2ubd4m/_remote_module_non_scriptable.py
2022-09-27T15:51:00.1719998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2jzj373
2022-09-27T15:51:00.1720828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2jzj373/_remote_module_non_scriptable.py
2022-09-27T15:51:00.3280099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:00.3281157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:00.3281955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:00.3282644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:00.3390907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:00.3464305Z STAGE:2022-09-27 15:51:00 7506:7506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:00.4293477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:00.4364588Z STAGE:2022-09-27 15:51:00 7505:7505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:00.4382139Z STAGE:2022-09-27 15:51:00 7505:7505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:00.4382771Z STAGE:2022-09-27 15:51:00 7506:7506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:00.4401445Z STAGE:2022-09-27 15:51:00 7505:7505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:00.4402399Z STAGE:2022-09-27 15:51:00 7506:7506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:00.4418127Z STAGE:2022-09-27 15:51:00 7506:7506 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:51:00 7505:7505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:00.4418538Z 
2022-09-27T15:51:00.4430117Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:00.4435676Z STAGE:2022-09-27 15:51:00 7506:7506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:00.4436802Z STAGE:2022-09-27 15:51:00 7505:7505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:00.4451081Z STAGE:2022-09-27 15:51:00 7506:7506 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:51:00 7505:7505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:00.4451495Z 
2022-09-27T15:51:00.4632896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7599
2022-09-27T15:51:00.4638571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7600
2022-09-27T15:51:00.9087433Z ok (4.470s)
2022-09-27T15:51:00.9087768Z 
2022-09-27T15:51:00.9088199Z ----------------------------------------------------------------------
2022-09-27T15:51:00.9088553Z Ran 1 test in 4.471s
2022-09-27T15:51:00.9088720Z 
2022-09-27T15:51:00.9088797Z OK
2022-09-27T15:51:00.9088936Z 
2022-09-27T15:51:00.9089072Z Generating XML reports...
2022-09-27T15:51:00.9124723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155056.xml
2022-09-27T15:51:02.1404570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:02.1405209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:02.1410335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:02.1410875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:02.2252736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:02.2253261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:02.2255847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:02.2256374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:02.4341796Z ok (7.597s)
2022-09-27T15:51:02.4342031Z 
2022-09-27T15:51:02.4343188Z ----------------------------------------------------------------------
2022-09-27T15:51:02.4343734Z Ran 1 test in 7.598s
2022-09-27T15:51:02.4343908Z 
2022-09-27T15:51:02.4343986Z OK
2022-09-27T15:51:02.4344137Z 
2022-09-27T15:51:02.4344273Z Generating XML reports...
2022-09-27T15:51:02.4379077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155054.xml
2022-09-27T15:51:02.6792135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:02.7774643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:02.7775526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:02.7805001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:02.7811739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:02.7878453Z STAGE:2022-09-27 15:51:02 7599:7599 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:02.8787616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:02.8852392Z STAGE:2022-09-27 15:51:02 7600:7600 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:03.0801932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:03.0802448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:03.0805001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:03.0805513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:03.5886371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:03.5909095Z 
2022-09-27T15:51:03.5909957Z Running tests...
2022-09-27T15:51:03.5911089Z ----------------------------------------------------------------------
2022-09-27T15:51:04.4683810Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:04.4684995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:04.4686213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:04.4687194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:04.5656486Z [1664293864.564961] [28fd6574525b:7599 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:51:04.5657687Z [1664293864.564961] [28fd6574525b:7599 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:51:04.6412462Z [1664293864.640483] [28fd6574525b:7600 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:51:04.6413624Z [1664293864.640483] [28fd6574525b:7600 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:51:04.9758046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:04.9781701Z 
2022-09-27T15:51:04.9782049Z Running tests...
2022-09-27T15:51:04.9782915Z ----------------------------------------------------------------------
2022-09-27T15:51:04.9801160Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:51:04.9801812Z 
2022-09-27T15:51:04.9802386Z ----------------------------------------------------------------------
2022-09-27T15:51:04.9803017Z Ran 1 test in 0.002s
2022-09-27T15:51:04.9803754Z 
2022-09-27T15:51:04.9803974Z OK (skipped=1)
2022-09-27T15:51:04.9804290Z 
2022-09-27T15:51:04.9804539Z Generating XML reports...
2022-09-27T15:51:04.9838153Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155104.xml
2022-09-27T15:51:05.0274954Z STAGE:2022-09-27 15:51:05 7599:7599 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:51:05 7600:7600 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:05.0275809Z 
2022-09-27T15:51:05.1500689Z STAGE:2022-09-27 15:51:05 7600:7600 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:51:05 7599:7599 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:05.1501718Z 
2022-09-27T15:51:05.1509233Z STAGE:2022-09-27 15:51:05 7600:7600 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:05.1510399Z STAGE:2022-09-27 15:51:05 7599:7599 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:05.1845863Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:05.2052435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7757
2022-09-27T15:51:05.2057211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7758
2022-09-27T15:51:06.4781176Z ok (7.689s)
2022-09-27T15:51:06.4781416Z 
2022-09-27T15:51:06.4781818Z ----------------------------------------------------------------------
2022-09-27T15:51:06.4782164Z Ran 1 test in 7.689s
2022-09-27T15:51:06.4782314Z 
2022-09-27T15:51:06.4782416Z OK
2022-09-27T15:51:06.4782555Z 
2022-09-27T15:51:06.4782690Z Generating XML reports...
2022-09-27T15:51:06.4837201Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155058.xml
2022-09-27T15:51:06.9695892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:06.9696440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:06.9699716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:06.9700208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:06.9857205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:06.9857714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:06.9861078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:06.9861579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:06.9978988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:06.9979899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:06.9980760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:06.9981257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:07.4858188Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:07.4881550Z 
2022-09-27T15:51:07.4882017Z Running tests...
2022-09-27T15:51:07.4882550Z ----------------------------------------------------------------------
2022-09-27T15:51:07.4901657Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-09-27T15:51:07.4902012Z 
2022-09-27T15:51:07.4902327Z ----------------------------------------------------------------------
2022-09-27T15:51:07.4902648Z Ran 1 test in 0.002s
2022-09-27T15:51:07.4902814Z 
2022-09-27T15:51:07.4902934Z OK (skipped=1)
2022-09-27T15:51:07.4903095Z 
2022-09-27T15:51:07.4903570Z Generating XML reports...
2022-09-27T15:51:07.4938855Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155107.xml
2022-09-27T15:51:07.5300441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:07.5301030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:07.5301799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:07.5302512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:07.5409276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:07.6361398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:07.6511353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:51:07.6516070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:51:07.6518338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:51:07.6616903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:51:07.6739154Z STAGE:2022-09-27 15:51:07 7758:7758 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:07.6762505Z STAGE:2022-09-27 15:51:07 7757:7757 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:07.6780112Z STAGE:2022-09-27 15:51:07 7758:7758 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:51:07 7757:7757 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:07.6780535Z 
2022-09-27T15:51:07.6797163Z STAGE:2022-09-27 15:51:07 7757:7757 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:07.6797751Z STAGE:2022-09-27 15:51:07 7758:7758 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:07.6811238Z STAGE:2022-09-27 15:51:07 7757:7757 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:07.6811835Z STAGE:2022-09-27 15:51:07 7758:7758 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:07.6825672Z STAGE:2022-09-27 15:51:07 7757:7757 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:07.6826258Z STAGE:2022-09-27 15:51:07 7758:7758 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:07.6838664Z STAGE:2022-09-27 15:51:07 7757:7757 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:07.6839292Z STAGE:2022-09-27 15:51:07 7758:7758 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:08.1125455Z ok (4.521s)
2022-09-27T15:51:08.1125661Z 
2022-09-27T15:51:08.1126060Z ----------------------------------------------------------------------
2022-09-27T15:51:08.1126408Z Ran 1 test in 4.522s
2022-09-27T15:51:08.1126581Z 
2022-09-27T15:51:08.1126681Z OK
2022-09-27T15:51:08.1126802Z 
2022-09-27T15:51:08.1126936Z Generating XML reports...
2022-09-27T15:51:08.1162264Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155103.xml
2022-09-27T15:51:08.6985992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:08.6986520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:08.6987491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:08.6987965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:09.1966535Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:09.1989926Z 
2022-09-27T15:51:09.1990351Z Running tests...
2022-09-27T15:51:09.1990861Z ----------------------------------------------------------------------
2022-09-27T15:51:09.2012382Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T15:51:09.2013422Z 
2022-09-27T15:51:09.2014067Z ----------------------------------------------------------------------
2022-09-27T15:51:09.2014828Z Ran 1 test in 0.002s
2022-09-27T15:51:09.2015190Z 
2022-09-27T15:51:09.2015446Z OK (skipped=1)
2022-09-27T15:51:09.2015880Z 
2022-09-27T15:51:09.2016148Z Generating XML reports...
2022-09-27T15:51:09.2063753Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155109.xml
2022-09-27T15:51:09.5631807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:09.5632358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:09.5634918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:09.5635424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:10.0448459Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:10.0470662Z 
2022-09-27T15:51:10.0470909Z Running tests...
2022-09-27T15:51:10.0471329Z ----------------------------------------------------------------------
2022-09-27T15:51:10.0489742Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-09-27T15:51:10.0490546Z 
2022-09-27T15:51:10.0490842Z ----------------------------------------------------------------------
2022-09-27T15:51:10.0491195Z Ran 1 test in 0.002s
2022-09-27T15:51:10.0491366Z 
2022-09-27T15:51:10.0491459Z OK (skipped=1)
2022-09-27T15:51:10.0491615Z 
2022-09-27T15:51:10.0491742Z Generating XML reports...
2022-09-27T15:51:10.0526036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155110.xml
2022-09-27T15:51:10.3340740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:10.3341243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:10.3343226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:10.3344028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:10.8362469Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:10.8385596Z 
2022-09-27T15:51:10.8385857Z Running tests...
2022-09-27T15:51:10.8386302Z ----------------------------------------------------------------------
2022-09-27T15:51:11.2713736Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:11.2714460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:11.2715072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:11.2715562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:11.7701976Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:11.7724677Z 
2022-09-27T15:51:11.7724941Z Running tests...
2022-09-27T15:51:11.7725641Z ----------------------------------------------------------------------
2022-09-27T15:51:11.7744618Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T15:51:11.7745769Z 
2022-09-27T15:51:11.7746196Z ----------------------------------------------------------------------
2022-09-27T15:51:11.7746518Z Ran 1 test in 0.002s
2022-09-27T15:51:11.7746689Z 
2022-09-27T15:51:11.7746798Z OK (skipped=1)
2022-09-27T15:51:11.7746954Z 
2022-09-27T15:51:11.7747080Z Generating XML reports...
2022-09-27T15:51:11.7780514Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155111.xml
2022-09-27T15:51:12.0712800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:12.0715583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:12.0716265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:12.0716762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:12.4699735Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:12.4897821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8042
2022-09-27T15:51:12.4904350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8043
2022-09-27T15:51:12.5678541Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:12.5701894Z 
2022-09-27T15:51:12.5702146Z Running tests...
2022-09-27T15:51:12.5702581Z ----------------------------------------------------------------------
2022-09-27T15:51:12.5724902Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-09-27T15:51:12.5725584Z 
2022-09-27T15:51:12.5726035Z ----------------------------------------------------------------------
2022-09-27T15:51:12.5726375Z Ran 1 test in 0.002s
2022-09-27T15:51:12.5726543Z 
2022-09-27T15:51:12.5726650Z OK (skipped=1)
2022-09-27T15:51:12.5726812Z 
2022-09-27T15:51:12.5726939Z Generating XML reports...
2022-09-27T15:51:12.5766900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155112.xml
2022-09-27T15:51:13.8577214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:13.8577734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:13.8578788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:13.8579305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:14.2283286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:14.2283843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:14.2287226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:14.2287738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:14.2310361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:14.2310831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:14.2314210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:14.2314696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:14.3606258Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:14.3629537Z 
2022-09-27T15:51:14.3629821Z Running tests...
2022-09-27T15:51:14.3630260Z ----------------------------------------------------------------------
2022-09-27T15:51:14.6006553Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:14.6007410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:14.6008426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:14.6008909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:14.7615928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:14.7616533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:14.7617320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:14.7618035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:14.7623901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:14.7624546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:15.1104142Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:15.1127192Z 
2022-09-27T15:51:15.1127496Z Running tests...
2022-09-27T15:51:15.1127950Z ----------------------------------------------------------------------
2022-09-27T15:51:15.1146027Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-09-27T15:51:15.1146374Z 
2022-09-27T15:51:15.1146674Z ----------------------------------------------------------------------
2022-09-27T15:51:15.1147000Z Ran 1 test in 0.002s
2022-09-27T15:51:15.1147169Z 
2022-09-27T15:51:15.1147285Z OK (skipped=1)
2022-09-27T15:51:15.1147450Z 
2022-09-27T15:51:15.1148936Z Generating XML reports...
2022-09-27T15:51:15.1182164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155115.xml
2022-09-27T15:51:15.1972591Z skip: Skipped due to small world size. (4.358s)
2022-09-27T15:51:15.1972836Z 
2022-09-27T15:51:15.1973219Z ----------------------------------------------------------------------
2022-09-27T15:51:15.1973566Z Ran 1 test in 4.359s
2022-09-27T15:51:15.1973738Z 
2022-09-27T15:51:15.1973854Z OK (skipped=1)
2022-09-27T15:51:15.1974024Z 
2022-09-27T15:51:15.1974136Z Generating XML reports...
2022-09-27T15:51:15.2010161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155110.xml
2022-09-27T15:51:15.9986977Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:16.0185580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8185
2022-09-27T15:51:16.0191788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8186
2022-09-27T15:51:17.1637319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:17.1637835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:17.1640146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:17.1640865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:17.4632711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:17.4633222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:17.4635549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:17.4636026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:17.6551659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:17.6574389Z 
2022-09-27T15:51:17.6574792Z Running tests...
2022-09-27T15:51:17.6575520Z ----------------------------------------------------------------------
2022-09-27T15:51:17.6601579Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-09-27T15:51:17.6601941Z 
2022-09-27T15:51:17.6602253Z ----------------------------------------------------------------------
2022-09-27T15:51:17.6602808Z Ran 1 test in 0.003s
2022-09-27T15:51:17.6602983Z 
2022-09-27T15:51:17.6603099Z OK (skipped=1)
2022-09-27T15:51:17.6603261Z 
2022-09-27T15:51:17.6603391Z Generating XML reports...
2022-09-27T15:51:17.6637493Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155117.xml
2022-09-27T15:51:17.7588021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:17.7588928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:17.7589993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:17.7590665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:17.8560118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:17.8561208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:17.8562170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:17.8562952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:17.9655391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:17.9678005Z 
2022-09-27T15:51:17.9678575Z Running tests...
2022-09-27T15:51:17.9679074Z ----------------------------------------------------------------------
2022-09-27T15:51:18.2701595Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:18.3682910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:18.3683724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:18.3715030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:18.3722129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:18.3725954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:51:18.4696312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:18.4698905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:51:18.4699929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:51:18.4740085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:51:18.4769096Z STAGE:2022-09-27 15:51:18 8186:8186 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:18.4801094Z STAGE:2022-09-27 15:51:18 8185:8185 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:19.6269179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:19.6270039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:19.6270874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:19.6271371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:19.6701773Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:19.6909803Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8360
2022-09-27T15:51:19.6915093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8361
2022-09-27T15:51:20.1424966Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:20.1447994Z 
2022-09-27T15:51:20.1448248Z Running tests...
2022-09-27T15:51:20.1448722Z ----------------------------------------------------------------------
2022-09-27T15:51:20.1466357Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:51:20.1466903Z 
2022-09-27T15:51:20.1467350Z ----------------------------------------------------------------------
2022-09-27T15:51:20.1467698Z Ran 1 test in 0.002s
2022-09-27T15:51:20.1467874Z 
2022-09-27T15:51:20.1467990Z OK (skipped=1)
2022-09-27T15:51:20.1468151Z 
2022-09-27T15:51:20.1468279Z Generating XML reports...
2022-09-27T15:51:20.1503577Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155120.xml
2022-09-27T15:51:20.1741008Z [1664293880.173648] [28fd6574525b:8185 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:51:20.1741581Z [1664293880.173648] [28fd6574525b:8185 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:51:20.1954404Z [1664293880.194966] [28fd6574525b:8186 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:51:20.1954948Z [1664293880.194966] [28fd6574525b:8186 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:51:20.6714373Z STAGE:2022-09-27 15:51:20 8185:8185 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:51:20 8186:8186 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:20.6714836Z 
2022-09-27T15:51:20.7964947Z STAGE:2022-09-27 15:51:20 8185:8185 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:20.7981615Z STAGE:2022-09-27 15:51:20 8186:8186 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:20.7992211Z STAGE:2022-09-27 15:51:20 8185:8185 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:20.7993040Z STAGE:2022-09-27 15:51:20 8186:8186 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:21.4230666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:21.4231246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:21.4231865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:21.4232337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:21.4557668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:21.4558196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:21.4561743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:21.4562457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:21.9924974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:21.9926325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:21.9927601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:21.9928983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:21.9933976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:21.9935092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:22.0030178Z STAGE:2022-09-27 15:51:22 8360:8360 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:51:22 8361:8361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:22.0030593Z 
2022-09-27T15:51:22.0047525Z STAGE:2022-09-27 15:51:22 8360:8360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:22.0048684Z STAGE:2022-09-27 15:51:22 8361:8361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:22.0063791Z STAGE:2022-09-27 15:51:22 8360:8360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:22.0064395Z STAGE:2022-09-27 15:51:22 8361:8361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:22.0078923Z STAGE:2022-09-27 15:51:22 8360:8360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:22.0080134Z STAGE:2022-09-27 15:51:22 8361:8361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:22.0093146Z STAGE:2022-09-27 15:51:22 8360:8360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:22.0095177Z STAGE:2022-09-27 15:51:22 8361:8361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:22.0107951Z STAGE:2022-09-27 15:51:22 8360:8360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:22.0109197Z STAGE:2022-09-27 15:51:22 8361:8361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:22.0319409Z ok (7.669s)
2022-09-27T15:51:22.0319770Z 
2022-09-27T15:51:22.0320890Z ----------------------------------------------------------------------
2022-09-27T15:51:22.0321597Z Ran 1 test in 7.669s
2022-09-27T15:51:22.0321947Z 
2022-09-27T15:51:22.0322131Z OK
2022-09-27T15:51:22.0322424Z 
2022-09-27T15:51:22.0322687Z Generating XML reports...
2022-09-27T15:51:22.0358370Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155114.xml
2022-09-27T15:51:22.1865168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:22.1866456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:22.1867551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:22.1868403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:22.3980900Z ok (4.430s)
2022-09-27T15:51:22.3981124Z 
2022-09-27T15:51:22.3981527Z ----------------------------------------------------------------------
2022-09-27T15:51:22.3981850Z Ran 1 test in 4.430s
2022-09-27T15:51:22.3982020Z 
2022-09-27T15:51:22.3982115Z OK
2022-09-27T15:51:22.3982256Z 
2022-09-27T15:51:22.3982387Z Generating XML reports...
2022-09-27T15:51:22.4017637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155117.xml
2022-09-27T15:51:22.6987688Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:22.7009861Z 
2022-09-27T15:51:22.7010302Z Running tests...
2022-09-27T15:51:22.7010826Z ----------------------------------------------------------------------
2022-09-27T15:51:24.2456259Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:24.2457616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:24.2458648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:24.2459121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:24.4044315Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:24.4235460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8543
2022-09-27T15:51:24.4241532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8544
2022-09-27T15:51:24.4641660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:24.4642152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:24.4645574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:24.4646066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:24.7512026Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:24.7535769Z 
2022-09-27T15:51:24.7536168Z Running tests...
2022-09-27T15:51:24.7536620Z ----------------------------------------------------------------------
2022-09-27T15:51:24.9480093Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:24.9502428Z 
2022-09-27T15:51:24.9502789Z Running tests...
2022-09-27T15:51:24.9503256Z ----------------------------------------------------------------------
2022-09-27T15:51:26.0830442Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:26.0831314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:26.0832621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:26.0833134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:26.0874251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:26.0874725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:26.0877057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:26.0877844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:26.4020650Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:26.4214535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8613
2022-09-27T15:51:26.4220680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8614
2022-09-27T15:51:26.5472872Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:26.5668838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8616
2022-09-27T15:51:26.5674693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8617
2022-09-27T15:51:26.6140247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:26.6145354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:26.6146188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:26.6241120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:26.6247917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:26.7161420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:28.1822011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:28.1822719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:28.1823884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:28.1824390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:28.2051277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:28.2051752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:28.2053430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:28.2054189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:28.3184341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:28.3185010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:28.3186189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:28.3186844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:28.3582294Z STAGE:2022-09-27 15:51:28 8543:8543 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:28.3603350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:28.3603823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:28.3607460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:28.3607961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:28.4140126Z STAGE:2022-09-27 15:51:28 8544:8544 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:28.7144342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:28.7287809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:28.7288621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:28.7347433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:28.7354361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:28.8301048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:28.9101536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:28.9102077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:28.9102866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:28.9103576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:28.9212526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:28.9283621Z STAGE:2022-09-27 15:51:28 8617:8617 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:29.0116428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:29.0190040Z STAGE:2022-09-27 15:51:29 8616:8616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:29.0205982Z STAGE:2022-09-27 15:51:29 8616:8616 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:29.0206700Z STAGE:2022-09-27 15:51:29 8617:8617 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:29.2287703Z skip: Skipped due to small world size. (4.475s)
2022-09-27T15:51:29.2287972Z 
2022-09-27T15:51:29.2288352Z ----------------------------------------------------------------------
2022-09-27T15:51:29.2288681Z Ran 1 test in 4.475s
2022-09-27T15:51:29.2288848Z 
2022-09-27T15:51:29.2288960Z OK (skipped=1)
2022-09-27T15:51:29.2289119Z 
2022-09-27T15:51:29.2289269Z Generating XML reports...
2022-09-27T15:51:29.2326067Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155124.xml
2022-09-27T15:51:29.4750369Z ok (4.524s)
2022-09-27T15:51:29.4750578Z 
2022-09-27T15:51:29.4751812Z ----------------------------------------------------------------------
2022-09-27T15:51:29.4752220Z Ran 1 test in 4.525s
2022-09-27T15:51:29.4752393Z 
2022-09-27T15:51:29.4752492Z OK
2022-09-27T15:51:29.4752628Z 
2022-09-27T15:51:29.4752750Z Generating XML reports...
2022-09-27T15:51:29.4787956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155124.xml
2022-09-27T15:51:29.8384585Z STAGE:2022-09-27 15:51:29 8543:8543 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:29.8387997Z STAGE:2022-09-27 15:51:29 8544:8544 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:30.0055015Z STAGE:2022-09-27 15:51:30 8544:8544 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:30.0059715Z STAGE:2022-09-27 15:51:30 8543:8543 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:30.0071568Z STAGE:2022-09-27 15:51:30 8544:8544 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:30.0077312Z STAGE:2022-09-27 15:51:30 8543:8543 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:30.8376040Z ok (8.136s)
2022-09-27T15:51:30.8376220Z 
2022-09-27T15:51:30.8376618Z ----------------------------------------------------------------------
2022-09-27T15:51:30.8376941Z Ran 1 test in 8.136s
2022-09-27T15:51:30.8377112Z 
2022-09-27T15:51:30.8377209Z OK
2022-09-27T15:51:30.8377346Z 
2022-09-27T15:51:30.8377481Z Generating XML reports...
2022-09-27T15:51:30.8412766Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155122.xml
2022-09-27T15:51:31.4004846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:31.4005455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:31.4006714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:31.4007205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:31.5866850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:31.5867377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:31.5870473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:31.5870960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:31.9484749Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:31.9522250Z 
2022-09-27T15:51:31.9523381Z Running tests...
2022-09-27T15:51:31.9524906Z ----------------------------------------------------------------------
2022-09-27T15:51:31.9555391Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.003s)
2022-09-27T15:51:31.9558529Z 
2022-09-27T15:51:31.9559265Z ----------------------------------------------------------------------
2022-09-27T15:51:31.9560036Z Ran 1 test in 0.003s
2022-09-27T15:51:31.9560967Z 
2022-09-27T15:51:31.9561221Z OK (skipped=1)
2022-09-27T15:51:31.9561586Z 
2022-09-27T15:51:31.9561843Z Generating XML reports...
2022-09-27T15:51:31.9613952Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155131.xml
2022-09-27T15:51:32.1313398Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:32.1336206Z 
2022-09-27T15:51:32.1336728Z Running tests...
2022-09-27T15:51:32.1337162Z ----------------------------------------------------------------------
2022-09-27T15:51:32.9967225Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:32.9969741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:32.9972801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:32.9973921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:33.5068653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:33.5092095Z 
2022-09-27T15:51:33.5092427Z Running tests...
2022-09-27T15:51:33.5092879Z ----------------------------------------------------------------------
2022-09-27T15:51:33.7283286Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:33.7478773Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8915
2022-09-27T15:51:33.7485276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8916
2022-09-27T15:51:33.8960859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:33.8961371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:33.8963526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:33.8964002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:34.3863446Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:34.3904670Z 
2022-09-27T15:51:34.3905699Z Running tests...
2022-09-27T15:51:34.3907210Z ----------------------------------------------------------------------
2022-09-27T15:51:34.3942110Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.004s)
2022-09-27T15:51:34.3943240Z 
2022-09-27T15:51:34.3943927Z ----------------------------------------------------------------------
2022-09-27T15:51:34.3944675Z Ran 1 test in 0.004s
2022-09-27T15:51:34.3945061Z 
2022-09-27T15:51:34.3945309Z OK (skipped=1)
2022-09-27T15:51:34.3945681Z 
2022-09-27T15:51:34.3945926Z Generating XML reports...
2022-09-27T15:51:34.4006056Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155134.xml
2022-09-27T15:51:35.1620946Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:35.1818934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8982
2022-09-27T15:51:35.1825396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8983
2022-09-27T15:51:35.4053105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:35.4058469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:35.4062108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:35.4063700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:35.5176150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:35.5176664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:35.5178625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:35.5179132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:36.1121687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:36.1122274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:36.1123075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:36.1123782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:36.1131206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:36.1131713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:36.1232018Z STAGE:2022-09-27 15:51:36 8915:8915 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:36.1232629Z STAGE:2022-09-27 15:51:36 8916:8916 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:36.1248265Z STAGE:2022-09-27 15:51:36 8916:8916 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:51:36 8915:8915 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:36.1248689Z 
2022-09-27T15:51:36.1277345Z STAGE:2022-09-27 15:51:36 8916:8916 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:51:36 8915:8915 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:36.1277767Z 
2022-09-27T15:51:36.1292548Z STAGE:2022-09-27 15:51:36 8916:8916 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:36.1293155Z STAGE:2022-09-27 15:51:36 8915:8915 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:36.4753713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:36.4754212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:36.4757866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:36.4758397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:36.5552915Z ok (4.421s)
2022-09-27T15:51:36.5553122Z 
2022-09-27T15:51:36.5553499Z ----------------------------------------------------------------------
2022-09-27T15:51:36.5553822Z Ran 1 test in 4.422s
2022-09-27T15:51:36.5553992Z 
2022-09-27T15:51:36.5554097Z OK
2022-09-27T15:51:36.5554235Z 
2022-09-27T15:51:36.5554376Z Generating XML reports...
2022-09-27T15:51:36.5590228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155132.xml
2022-09-27T15:51:36.8579113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:36.8579649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:36.8580874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:36.8581372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:36.8793237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:36.8793739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:36.8796011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:36.8796507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:36.9812748Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:36.9836181Z 
2022-09-27T15:51:36.9836441Z Running tests...
2022-09-27T15:51:36.9836890Z ----------------------------------------------------------------------
2022-09-27T15:51:36.9860147Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T15:51:36.9860652Z 
2022-09-27T15:51:36.9860919Z ----------------------------------------------------------------------
2022-09-27T15:51:36.9861287Z Ran 1 test in 0.002s
2022-09-27T15:51:36.9861462Z 
2022-09-27T15:51:36.9861572Z OK (skipped=1)
2022-09-27T15:51:36.9861732Z 
2022-09-27T15:51:36.9861863Z Generating XML reports...
2022-09-27T15:51:36.9902684Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155136.xml
2022-09-27T15:51:37.3664511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:37.3934790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:37.3935641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:37.3968533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:37.3975307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:37.4948616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:38.7129766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:38.7130766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:38.7131944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:38.7132880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:38.9594874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:38.9595870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:38.9597040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:38.9597966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:39.0751663Z STAGE:2022-09-27 15:51:39 8983:8983 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:39.1557702Z STAGE:2022-09-27 15:51:39 8982:8982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:39.2348114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:39.2371827Z 
2022-09-27T15:51:39.2372289Z Running tests...
2022-09-27T15:51:39.2372785Z ----------------------------------------------------------------------
2022-09-27T15:51:39.2391492Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T15:51:39.2391826Z 
2022-09-27T15:51:39.2392116Z ----------------------------------------------------------------------
2022-09-27T15:51:39.2392448Z Ran 1 test in 0.002s
2022-09-27T15:51:39.2392898Z 
2022-09-27T15:51:39.2393015Z OK (skipped=1)
2022-09-27T15:51:39.2393171Z 
2022-09-27T15:51:39.2394395Z Generating XML reports...
2022-09-27T15:51:39.2428384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155139.xml
2022-09-27T15:51:39.4735739Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:39.4758208Z 
2022-09-27T15:51:39.4758622Z Running tests...
2022-09-27T15:51:39.4759149Z ----------------------------------------------------------------------
2022-09-27T15:51:39.4779193Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T15:51:39.4779738Z 
2022-09-27T15:51:39.4780119Z ----------------------------------------------------------------------
2022-09-27T15:51:39.4780442Z Ran 1 test in 0.002s
2022-09-27T15:51:39.4780608Z 
2022-09-27T15:51:39.4780729Z OK (skipped=1)
2022-09-27T15:51:39.4780886Z 
2022-09-27T15:51:39.4781013Z Generating XML reports...
2022-09-27T15:51:39.4814886Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155139.xml
2022-09-27T15:51:40.5836457Z STAGE:2022-09-27 15:51:40 8982:8982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:40.6142021Z STAGE:2022-09-27 15:51:40 8983:8983 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:40.7683156Z STAGE:2022-09-27 15:51:40 8982:8982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:40.7699276Z STAGE:2022-09-27 15:51:40 8982:8982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:40.7831070Z STAGE:2022-09-27 15:51:40 8983:8983 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:40.7849079Z STAGE:2022-09-27 15:51:40 8983:8983 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:41.3068961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:41.3069508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:41.3071939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:41.3072436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:41.4959535Z ok (7.986s)
2022-09-27T15:51:41.4959969Z 
2022-09-27T15:51:41.4961040Z ----------------------------------------------------------------------
2022-09-27T15:51:41.4961705Z Ran 1 test in 7.987s
2022-09-27T15:51:41.4962018Z 
2022-09-27T15:51:41.4962189Z OK
2022-09-27T15:51:41.4962429Z 
2022-09-27T15:51:41.4963018Z Generating XML reports...
2022-09-27T15:51:41.4999979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155133.xml
2022-09-27T15:51:41.5165706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:41.5166534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:41.5168517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:41.5169325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:41.8289973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:41.8312707Z 
2022-09-27T15:51:41.8313183Z Running tests...
2022-09-27T15:51:41.8313983Z ----------------------------------------------------------------------
2022-09-27T15:51:41.8333437Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T15:51:41.8334102Z 
2022-09-27T15:51:41.8334648Z ----------------------------------------------------------------------
2022-09-27T15:51:41.8335657Z Ran 1 test in 0.002s
2022-09-27T15:51:41.8335957Z 
2022-09-27T15:51:41.8336150Z OK (skipped=1)
2022-09-27T15:51:41.8336447Z 
2022-09-27T15:51:41.8336659Z Generating XML reports...
2022-09-27T15:51:41.8370139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155141.xml
2022-09-27T15:51:42.0101166Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:42.0123514Z 
2022-09-27T15:51:42.0123783Z Running tests...
2022-09-27T15:51:42.0124220Z ----------------------------------------------------------------------
2022-09-27T15:51:42.0148501Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-09-27T15:51:42.0149138Z 
2022-09-27T15:51:42.0149433Z ----------------------------------------------------------------------
2022-09-27T15:51:42.0149773Z Ran 1 test in 0.003s
2022-09-27T15:51:42.0149954Z 
2022-09-27T15:51:42.0150047Z OK (skipped=1)
2022-09-27T15:51:42.0150205Z 
2022-09-27T15:51:42.0150332Z Generating XML reports...
2022-09-27T15:51:42.0184663Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155142.xml
2022-09-27T15:51:43.7447140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:43.7447659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:43.7448988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:43.7449459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:43.8507011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:43.8507512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:43.8508841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:43.8509308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:44.0427469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:44.0428046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:44.0430267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:44.0430764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:44.2394310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:44.2417871Z 
2022-09-27T15:51:44.2418126Z Running tests...
2022-09-27T15:51:44.2418555Z ----------------------------------------------------------------------
2022-09-27T15:51:44.2436480Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:51:44.2436799Z 
2022-09-27T15:51:44.2437078Z ----------------------------------------------------------------------
2022-09-27T15:51:44.2437395Z Ran 1 test in 0.002s
2022-09-27T15:51:44.2437559Z 
2022-09-27T15:51:44.2437668Z OK (skipped=1)
2022-09-27T15:51:44.2437824Z 
2022-09-27T15:51:44.2437953Z Generating XML reports...
2022-09-27T15:51:44.2472185Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155144.xml
2022-09-27T15:51:44.3405745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:44.3428536Z 
2022-09-27T15:51:44.3428830Z Running tests...
2022-09-27T15:51:44.3429282Z ----------------------------------------------------------------------
2022-09-27T15:51:44.5310676Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:44.5333887Z 
2022-09-27T15:51:44.5334258Z Running tests...
2022-09-27T15:51:44.5334753Z ----------------------------------------------------------------------
2022-09-27T15:51:44.5352600Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:51:44.5353562Z 
2022-09-27T15:51:44.5353844Z ----------------------------------------------------------------------
2022-09-27T15:51:44.5354168Z Ran 1 test in 0.002s
2022-09-27T15:51:44.5354331Z 
2022-09-27T15:51:44.5354440Z OK (skipped=1)
2022-09-27T15:51:44.5354602Z 
2022-09-27T15:51:44.5354723Z Generating XML reports...
2022-09-27T15:51:44.5388354Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155144.xml
2022-09-27T15:51:45.9668335Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:45.9861477Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9412
2022-09-27T15:51:45.9867716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9413
2022-09-27T15:51:46.2108288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:46.2108789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:46.2111509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:46.2112007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:46.6051013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:46.6053973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:46.6055690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:46.6056858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:46.7567590Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:46.7589407Z 
2022-09-27T15:51:46.7589755Z Running tests...
2022-09-27T15:51:46.7590257Z ----------------------------------------------------------------------
2022-09-27T15:51:46.7607904Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:51:46.7608240Z 
2022-09-27T15:51:46.7608561Z ----------------------------------------------------------------------
2022-09-27T15:51:46.7608897Z Ran 1 test in 0.002s
2022-09-27T15:51:46.7609064Z 
2022-09-27T15:51:46.7609450Z OK (skipped=1)
2022-09-27T15:51:46.7609633Z 
2022-09-27T15:51:46.7609762Z Generating XML reports...
2022-09-27T15:51:46.7642302Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155146.xml
2022-09-27T15:51:47.1651866Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:47.1675007Z 
2022-09-27T15:51:47.1675243Z Running tests...
2022-09-27T15:51:47.1675688Z ----------------------------------------------------------------------
2022-09-27T15:51:47.1695094Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:51:47.1695472Z 
2022-09-27T15:51:47.1695756Z ----------------------------------------------------------------------
2022-09-27T15:51:47.1696077Z Ran 1 test in 0.002s
2022-09-27T15:51:47.1696248Z 
2022-09-27T15:51:47.1696358Z OK (skipped=1)
2022-09-27T15:51:47.1696531Z 
2022-09-27T15:51:47.1696660Z Generating XML reports...
2022-09-27T15:51:47.1731489Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155147.xml
2022-09-27T15:51:47.6019536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:47.6020146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:47.6021068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:47.6021566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:47.6748418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:47.6749614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:47.6753552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:47.6754740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:48.2238215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:48.2240019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:48.2242390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:48.2244062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:48.2251490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:48.2254654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:48.2465717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:51:48.2467367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:51:48.2469308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:51:48.2477097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:51:48.2585575Z STAGE:2022-09-27 15:51:48 9412:9412 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:48.2702898Z STAGE:2022-09-27 15:51:48 9413:9413 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:48.2721796Z STAGE:2022-09-27 15:51:48 9412:9412 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:48.2722412Z STAGE:2022-09-27 15:51:48 9413:9413 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:48.2749368Z STAGE:2022-09-27 15:51:48 9412:9412 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:48.2750024Z STAGE:2022-09-27 15:51:48 9413:9413 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:51:48.2765389Z STAGE:2022-09-27 15:51:48 9412:9412 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:48.2765995Z STAGE:2022-09-27 15:51:48 9413:9413 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:51:48.6933115Z ok (4.350s)
2022-09-27T15:51:48.6933468Z 
2022-09-27T15:51:48.6934238Z ----------------------------------------------------------------------
2022-09-27T15:51:48.6934797Z Ran 1 test in 4.350s
2022-09-27T15:51:48.6934972Z 
2022-09-27T15:51:48.6935067Z OK
2022-09-27T15:51:48.6935201Z 
2022-09-27T15:51:48.6935346Z Generating XML reports...
2022-09-27T15:51:48.6970781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155144.xml
2022-09-27T15:51:48.7261681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:48.7262534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:48.7264493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:48.7265265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:49.2030660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:49.2031232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:49.2032543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:49.2033335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:49.2195532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:49.2219203Z 
2022-09-27T15:51:49.2219546Z Running tests...
2022-09-27T15:51:49.2220533Z ----------------------------------------------------------------------
2022-09-27T15:51:49.7113836Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:49.7136993Z 
2022-09-27T15:51:49.7137586Z Running tests...
2022-09-27T15:51:49.7138476Z ----------------------------------------------------------------------
2022-09-27T15:51:49.7156129Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:51:49.7156921Z 
2022-09-27T15:51:49.7157452Z ----------------------------------------------------------------------
2022-09-27T15:51:49.7157805Z Ran 1 test in 0.002s
2022-09-27T15:51:49.7157971Z 
2022-09-27T15:51:49.7158095Z OK (skipped=1)
2022-09-27T15:51:49.7158250Z 
2022-09-27T15:51:49.7158360Z Generating XML reports...
2022-09-27T15:51:49.7193778Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155149.xml
2022-09-27T15:51:50.7158094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:50.7159070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:50.7160813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:50.7161768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:50.7683393Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:50.7848424Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.562s)
2022-09-27T15:51:50.7849611Z 
2022-09-27T15:51:50.7850198Z ----------------------------------------------------------------------
2022-09-27T15:51:50.7850867Z Ran 1 test in 1.563s
2022-09-27T15:51:50.7851183Z 
2022-09-27T15:51:50.7851381Z OK (skipped=1)
2022-09-27T15:51:50.7851674Z 
2022-09-27T15:51:50.7851887Z Generating XML reports...
2022-09-27T15:51:50.7885528Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155149.xml
2022-09-27T15:51:51.2122317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:51.2146015Z 
2022-09-27T15:51:51.2146608Z Running tests...
2022-09-27T15:51:51.2147488Z ----------------------------------------------------------------------
2022-09-27T15:51:51.6528745Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:51.6529912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:51.6531551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:51.6532490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:52.2233809Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:52.2257394Z 
2022-09-27T15:51:52.2257728Z Running tests...
2022-09-27T15:51:52.2258554Z ----------------------------------------------------------------------
2022-09-27T15:51:52.2277416Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:51:52.2278110Z 
2022-09-27T15:51:52.2278656Z ----------------------------------------------------------------------
2022-09-27T15:51:52.2279297Z Ran 1 test in 0.002s
2022-09-27T15:51:52.2279617Z 
2022-09-27T15:51:52.2279803Z OK (skipped=1)
2022-09-27T15:51:52.2280107Z 
2022-09-27T15:51:52.2280660Z Generating XML reports...
2022-09-27T15:51:52.2314974Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155152.xml
2022-09-27T15:51:52.7560007Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:52.7757450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9666
2022-09-27T15:51:52.7763013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9667
2022-09-27T15:51:52.7941964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:52.7942575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:52.7944411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:52.7945249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:53.3440019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:53.3462973Z 
2022-09-27T15:51:53.3463233Z Running tests...
2022-09-27T15:51:53.3463698Z ----------------------------------------------------------------------
2022-09-27T15:51:54.2366405Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:54.2367007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:54.2369723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:54.2370220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:54.4599783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:54.4600558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:54.4601969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:54.4602453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:54.4688723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:54.4689179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:54.4692243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:54.4692728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:54.7523886Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:54.7548177Z 
2022-09-27T15:51:54.7548693Z Running tests...
2022-09-27T15:51:54.7549354Z ----------------------------------------------------------------------
2022-09-27T15:51:54.7568783Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:51:54.7569237Z 
2022-09-27T15:51:54.7569520Z ----------------------------------------------------------------------
2022-09-27T15:51:54.7569861Z Ran 1 test in 0.002s
2022-09-27T15:51:54.7570150Z 
2022-09-27T15:51:54.7570347Z OK (skipped=1)
2022-09-27T15:51:54.7570526Z 
2022-09-27T15:51:54.7570637Z Generating XML reports...
2022-09-27T15:51:54.7606315Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155154.xml
2022-09-27T15:51:54.9410667Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:54.9582368Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-09-27T15:51:54.9583246Z 
2022-09-27T15:51:54.9583536Z ----------------------------------------------------------------------
2022-09-27T15:51:54.9583857Z Ran 1 test in 1.612s
2022-09-27T15:51:54.9584100Z 
2022-09-27T15:51:54.9584304Z OK (skipped=1)
2022-09-27T15:51:54.9584538Z 
2022-09-27T15:51:54.9584667Z Generating XML reports...
2022-09-27T15:51:54.9617846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155153.xml
2022-09-27T15:51:55.0114552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:51:55.0115146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:51:55.0115955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:55.0116668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:51:55.0225587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:51:55.1129318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:51:55.4825769Z skip: Skipped due to small world size. (4.268s)
2022-09-27T15:51:55.4826038Z 
2022-09-27T15:51:55.4826437Z ----------------------------------------------------------------------
2022-09-27T15:51:55.4826762Z Ran 1 test in 4.268s
2022-09-27T15:51:55.4826931Z 
2022-09-27T15:51:55.4827041Z OK (skipped=1)
2022-09-27T15:51:55.4827202Z 
2022-09-27T15:51:55.4827328Z Generating XML reports...
2022-09-27T15:51:55.4869778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155151.xml
2022-09-27T15:51:56.8284529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:56.8285517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:56.8286762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:56.8287707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:57.2231689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:57.2232719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:57.2233853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:57.2234837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:57.3382171Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:57.3406137Z 
2022-09-27T15:51:57.3406860Z Running tests...
2022-09-27T15:51:57.3407664Z ----------------------------------------------------------------------
2022-09-27T15:51:57.3427580Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:51:57.3428271Z 
2022-09-27T15:51:57.3428805Z ----------------------------------------------------------------------
2022-09-27T15:51:57.3429437Z Ran 1 test in 0.002s
2022-09-27T15:51:57.3429740Z 
2022-09-27T15:51:57.3429952Z OK (skipped=1)
2022-09-27T15:51:57.3430252Z 
2022-09-27T15:51:57.3430490Z Generating XML reports...
2022-09-27T15:51:57.3465276Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155157.xml
2022-09-27T15:51:57.6297101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:57.6298065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:57.6299673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:57.6300219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:57.7331252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:51:57.7355285Z 
2022-09-27T15:51:57.7355751Z Running tests...
2022-09-27T15:51:57.7356238Z ----------------------------------------------------------------------
2022-09-27T15:51:58.1258531Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:51:58.1280735Z 
2022-09-27T15:51:58.1281779Z Running tests...
2022-09-27T15:51:58.1282384Z ----------------------------------------------------------------------
2022-09-27T15:51:58.1300652Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T15:51:58.1301269Z 
2022-09-27T15:51:58.1301572Z ----------------------------------------------------------------------
2022-09-27T15:51:58.1301912Z Ran 1 test in 0.002s
2022-09-27T15:51:58.1302073Z 
2022-09-27T15:51:58.1302182Z OK (skipped=1)
2022-09-27T15:51:58.1302336Z 
2022-09-27T15:51:58.1302446Z Generating XML reports...
2022-09-27T15:51:58.1336470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155158.xml
2022-09-27T15:51:59.3227028Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:51:59.3305714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:51:59.3307093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:51:59.3309287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:51:59.3310264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:51:59.3427898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9940
2022-09-27T15:51:59.3435930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9941
2022-09-27T15:51:59.8330506Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:51:59.8353922Z 
2022-09-27T15:51:59.8354127Z Running tests...
2022-09-27T15:51:59.8354801Z ----------------------------------------------------------------------
2022-09-27T15:51:59.8375042Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:51:59.8375440Z 
2022-09-27T15:51:59.8375748Z ----------------------------------------------------------------------
2022-09-27T15:51:59.8376088Z Ran 1 test in 0.002s
2022-09-27T15:51:59.8376237Z 
2022-09-27T15:51:59.8376347Z OK (skipped=1)
2022-09-27T15:51:59.8376832Z 
2022-09-27T15:51:59.8376961Z Generating XML reports...
2022-09-27T15:51:59.8411398Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155159.xml
2022-09-27T15:52:00.1609734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:00.1610728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:00.1611901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:00.1612855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:00.7021991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:00.7046231Z 
2022-09-27T15:52:00.7046647Z Running tests...
2022-09-27T15:52:00.7047168Z ----------------------------------------------------------------------
2022-09-27T15:52:00.7067365Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T15:52:00.7067942Z 
2022-09-27T15:52:00.7068366Z ----------------------------------------------------------------------
2022-09-27T15:52:00.7068851Z Ran 1 test in 0.002s
2022-09-27T15:52:00.7069020Z 
2022-09-27T15:52:00.7069131Z OK (skipped=1)
2022-09-27T15:52:00.7069288Z 
2022-09-27T15:52:00.7069419Z Generating XML reports...
2022-09-27T15:52:00.7104744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155200.xml
2022-09-27T15:52:01.0270930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:01.0271962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:01.0273177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:01.0274143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:01.0681530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:01.0682526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:01.0684073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:01.0685048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:01.5372232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:01.5907147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:01.5908753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:01.5982611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:01.5991224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:01.6964843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:01.9235256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:01.9236219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:01.9237386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:01.9238373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:02.4480082Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:02.4505533Z 
2022-09-27T15:52:02.4506071Z Running tests...
2022-09-27T15:52:02.4506946Z ----------------------------------------------------------------------
2022-09-27T15:52:02.4526580Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:02.4527233Z 
2022-09-27T15:52:02.4528123Z ----------------------------------------------------------------------
2022-09-27T15:52:02.4528735Z Ran 1 test in 0.002s
2022-09-27T15:52:02.4529046Z 
2022-09-27T15:52:02.4529248Z OK (skipped=1)
2022-09-27T15:52:02.4529560Z 
2022-09-27T15:52:02.4529798Z Generating XML reports...
2022-09-27T15:52:02.4566227Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155202.xml
2022-09-27T15:52:02.6410129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:02.6411202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:02.6412406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:02.6413393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:03.1578406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:03.1600400Z 
2022-09-27T15:52:03.1600640Z Running tests...
2022-09-27T15:52:03.1601318Z ----------------------------------------------------------------------
2022-09-27T15:52:04.5799632Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:04.5800524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:04.5801646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:04.5802182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:04.7782765Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:04.7974549Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10124
2022-09-27T15:52:04.7981607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10125
2022-09-27T15:52:04.8555481Z ok (7.120s)
2022-09-27T15:52:04.8555791Z 
2022-09-27T15:52:04.8556571Z ----------------------------------------------------------------------
2022-09-27T15:52:04.8557101Z Ran 1 test in 7.120s
2022-09-27T15:52:04.8557269Z 
2022-09-27T15:52:04.8557345Z OK
2022-09-27T15:52:04.8557480Z 
2022-09-27T15:52:04.8557608Z Generating XML reports...
2022-09-27T15:52:04.8594193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155157.xml
2022-09-27T15:52:05.1138980Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:05.1162230Z 
2022-09-27T15:52:05.1162441Z Running tests...
2022-09-27T15:52:05.1163149Z ----------------------------------------------------------------------
2022-09-27T15:52:05.1182168Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:05.1182754Z 
2022-09-27T15:52:05.1183325Z ----------------------------------------------------------------------
2022-09-27T15:52:05.1183997Z Ran 1 test in 0.002s
2022-09-27T15:52:05.1184238Z 
2022-09-27T15:52:05.1184348Z OK (skipped=1)
2022-09-27T15:52:05.1184511Z 
2022-09-27T15:52:05.1184636Z Generating XML reports...
2022-09-27T15:52:05.1219380Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155205.xml
2022-09-27T15:52:06.5698482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:06.5699858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:06.5701047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:06.5701996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:06.5871674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:06.5872607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:06.5874911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:06.5875870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:06.9827389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:06.9828365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:06.9829563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:06.9830499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:07.1297743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:07.1298755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:07.1299996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:07.1301266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:07.1306758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:07.1308545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:07.2390081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:07.2390960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:07.2391944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:07.2392757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:07.4883737Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:07.4906523Z 
2022-09-27T15:52:07.4906662Z Running tests...
2022-09-27T15:52:07.4907355Z ----------------------------------------------------------------------
2022-09-27T15:52:07.5059795Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... ok (4.346s)
2022-09-27T15:52:07.5060137Z 
2022-09-27T15:52:07.5060447Z ----------------------------------------------------------------------
2022-09-27T15:52:07.5060803Z Ran 1 test in 4.346s
2022-09-27T15:52:07.5060972Z 
2022-09-27T15:52:07.5061074Z OK
2022-09-27T15:52:07.5061211Z 
2022-09-27T15:52:07.5061338Z Generating XML reports...
2022-09-27T15:52:07.5116095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155203.xml
2022-09-27T15:52:07.7435010Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:07.7457910Z 
2022-09-27T15:52:07.7458543Z Running tests...
2022-09-27T15:52:07.7459542Z ----------------------------------------------------------------------
2022-09-27T15:52:09.2175307Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:09.2369672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10299
2022-09-27T15:52:09.2376087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10300
2022-09-27T15:52:09.3740696Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:09.3934765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10302
2022-09-27T15:52:09.3940675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10303
2022-09-27T15:52:09.6506886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:09.6507598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:09.6508588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:09.6509077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:10.2836394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:10.2859827Z 
2022-09-27T15:52:10.2860336Z Running tests...
2022-09-27T15:52:10.2860833Z ----------------------------------------------------------------------
2022-09-27T15:52:10.9168131Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:10.9169087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:10.9169968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:10.9170445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:10.9632980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:10.9633637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:10.9634952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:10.9635435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:11.1401972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:11.1402834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:11.1404056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:11.1404531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:11.1413333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:11.1414101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:11.1417812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:11.1418304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:11.4438765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:11.4854084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:11.4854920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:11.4945581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:11.4952604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:11.5867344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:11.6074007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:11.6074838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:11.6075607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:11.6076296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:11.6672853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:11.6806252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:11.6807082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:11.6875744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:11.6882847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:11.7821083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:11.9070682Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:11.9285264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10455
2022-09-27T15:52:11.9290942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10456
2022-09-27T15:52:12.2017997Z ok (4.456s)
2022-09-27T15:52:12.2018349Z 
2022-09-27T15:52:12.2018777Z ----------------------------------------------------------------------
2022-09-27T15:52:12.2019111Z Ran 1 test in 4.456s
2022-09-27T15:52:12.2019282Z 
2022-09-27T15:52:12.2019377Z OK
2022-09-27T15:52:12.2019517Z 
2022-09-27T15:52:12.2019675Z Generating XML reports...
2022-09-27T15:52:12.2054678Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155207.xml
2022-09-27T15:52:13.5998076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:13.5998593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:13.5999541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:13.6000014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:13.6576456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:13.6576957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:13.6578620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:13.6579134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:14.1898410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:14.1899490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:14.1900940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:14.1902350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:14.1906596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:14.1907556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:14.2226213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:14.2324445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:14.2326295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:14.2327714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:14.2475234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T15:52:14.2476540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T15:52:14.2477328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T15:52:14.2576931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T15:52:14.2698061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-09-27T15:52:14.2699241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-09-27T15:52:14.2700740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-09-27T15:52:14.2702212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-09-27T15:52:14.3632531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:14.3633503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:14.3634642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:14.3635606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:14.7363449Z ok (4.450s)
2022-09-27T15:52:14.7363837Z 
2022-09-27T15:52:14.7364514Z ----------------------------------------------------------------------
2022-09-27T15:52:14.7365167Z Ran 1 test in 4.450s
2022-09-27T15:52:14.7365494Z 
2022-09-27T15:52:14.7365671Z OK
2022-09-27T15:52:14.7365931Z 
2022-09-27T15:52:14.7366171Z Generating XML reports...
2022-09-27T15:52:14.7401765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155210.xml
2022-09-27T15:52:14.8824568Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:14.8849477Z 
2022-09-27T15:52:14.8850003Z Running tests...
2022-09-27T15:52:14.8850539Z ----------------------------------------------------------------------
2022-09-27T15:52:14.8870109Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:14.8870914Z 
2022-09-27T15:52:14.8871337Z ----------------------------------------------------------------------
2022-09-27T15:52:14.8871659Z Ran 1 test in 0.002s
2022-09-27T15:52:14.8871839Z 
2022-09-27T15:52:14.8871954Z OK (skipped=1)
2022-09-27T15:52:14.8872242Z 
2022-09-27T15:52:14.8872474Z Generating XML reports...
2022-09-27T15:52:14.8907587Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155214.xml
2022-09-27T15:52:15.3696565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T15:52:15.3697136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T15:52:15.3697938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T15:52:15.3698661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T15:52:15.4452328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-09-27T15:52:15.4452911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-09-27T15:52:15.4453984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-09-27T15:52:15.4454691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-09-27T15:52:16.2522798Z ok (8.761s)
2022-09-27T15:52:16.2523016Z 
2022-09-27T15:52:16.2523448Z ----------------------------------------------------------------------
2022-09-27T15:52:16.2523807Z Ran 1 test in 8.761s
2022-09-27T15:52:16.2523976Z 
2022-09-27T15:52:16.2524069Z OK
2022-09-27T15:52:16.2524203Z 
2022-09-27T15:52:16.2524341Z Generating XML reports...
2022-09-27T15:52:16.2560764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155207.xml
2022-09-27T15:52:16.8767635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:16.8768179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:16.8770044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:16.8770523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:16.9006332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:16.9006810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:16.9010799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:16.9011288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:17.4268302Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:17.4291531Z 
2022-09-27T15:52:17.4292023Z Running tests...
2022-09-27T15:52:17.4292467Z ----------------------------------------------------------------------
2022-09-27T15:52:17.4310703Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:17.4311058Z 
2022-09-27T15:52:17.4311327Z ----------------------------------------------------------------------
2022-09-27T15:52:17.4311662Z Ran 1 test in 0.002s
2022-09-27T15:52:17.4311827Z 
2022-09-27T15:52:17.4311936Z OK (skipped=1)
2022-09-27T15:52:17.4312094Z 
2022-09-27T15:52:17.4312203Z Generating XML reports...
2022-09-27T15:52:17.4346960Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155217.xml
2022-09-27T15:52:17.4652169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:17.4675978Z 
2022-09-27T15:52:17.4676569Z Running tests...
2022-09-27T15:52:17.4687297Z ----------------------------------------------------------------------
2022-09-27T15:52:17.4703355Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.003s)
2022-09-27T15:52:17.4703789Z 
2022-09-27T15:52:17.4704333Z ----------------------------------------------------------------------
2022-09-27T15:52:17.4705084Z Ran 1 test in 0.003s
2022-09-27T15:52:17.4705374Z 
2022-09-27T15:52:17.4705542Z OK (skipped=1)
2022-09-27T15:52:17.4705704Z 
2022-09-27T15:52:17.4705833Z Generating XML reports...
2022-09-27T15:52:17.4742026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155217.xml
2022-09-27T15:52:18.3381638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:18.3382166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:18.3384323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:18.3385506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:18.9287002Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:18.9308910Z 
2022-09-27T15:52:18.9309334Z Running tests...
2022-09-27T15:52:18.9310238Z ----------------------------------------------------------------------
2022-09-27T15:52:19.4050567Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:19.4051159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:19.4052636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:19.4053110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:19.4399074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:19.4399579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:19.4401916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:19.4402416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:19.8930300Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:19.8952435Z 
2022-09-27T15:52:19.8952721Z Running tests...
2022-09-27T15:52:19.8953169Z ----------------------------------------------------------------------
2022-09-27T15:52:19.8972059Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:19.8972386Z 
2022-09-27T15:52:19.8972663Z ----------------------------------------------------------------------
2022-09-27T15:52:19.8973014Z Ran 1 test in 0.002s
2022-09-27T15:52:19.8973183Z 
2022-09-27T15:52:19.8973303Z OK (skipped=1)
2022-09-27T15:52:19.8973443Z 
2022-09-27T15:52:19.8973573Z Generating XML reports...
2022-09-27T15:52:19.9007708Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155219.xml
2022-09-27T15:52:19.9371406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:19.9393476Z 
2022-09-27T15:52:19.9393995Z Running tests...
2022-09-27T15:52:19.9394481Z ----------------------------------------------------------------------
2022-09-27T15:52:20.5174794Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:20.5370467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10793
2022-09-27T15:52:20.5376149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10794
2022-09-27T15:52:21.6731373Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:21.6932002Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10889
2022-09-27T15:52:21.6938100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10890
2022-09-27T15:52:21.9695343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:21.9696303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:21.9697506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:21.9698452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:22.2670697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:22.2671197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:22.2672353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:22.2672848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:22.3295284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:22.3297283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:22.3303324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:22.3305073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:22.5258480Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:22.5281069Z 
2022-09-27T15:52:22.5281458Z Running tests...
2022-09-27T15:52:22.5282132Z ----------------------------------------------------------------------
2022-09-27T15:52:22.8268637Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:22.8959222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:22.8960070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:22.8978003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:22.8985602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:22.9972668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:23.3076945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:23.3077954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:23.3079098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:23.3080063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:23.3112124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:23.3113050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:23.3116107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:23.3117444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:23.8686925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:23.8687993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:23.8689149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:23.8690506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:23.8695956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:23.8697741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:23.8808865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:23.8809950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:23.8811530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:23.8813236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:23.8914858Z STAGE:2022-09-27 15:52:23 10889:10889 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:23.8915489Z STAGE:2022-09-27 15:52:23 10890:10890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:23.8932223Z STAGE:2022-09-27 15:52:23 10890:10890 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:23 10889:10889 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:23.8932961Z 
2022-09-27T15:52:23.8957764Z STAGE:2022-09-27 15:52:23 10889:10889 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:23.8958415Z STAGE:2022-09-27 15:52:23 10890:10890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:23.8974568Z STAGE:2022-09-27 15:52:23 10889:10889 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:23.8975814Z STAGE:2022-09-27 15:52:23 10890:10890 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:24.1614701Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:24.1831672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10986
2022-09-27T15:52:24.1836118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10987
2022-09-27T15:52:24.4011316Z ok (4.461s)
2022-09-27T15:52:24.4011683Z 
2022-09-27T15:52:24.4012174Z ----------------------------------------------------------------------
2022-09-27T15:52:24.4012505Z Ran 1 test in 4.462s
2022-09-27T15:52:24.4012674Z 
2022-09-27T15:52:24.4012773Z OK
2022-09-27T15:52:24.4012913Z 
2022-09-27T15:52:24.4013069Z Generating XML reports...
2022-09-27T15:52:24.4048506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155219.xml
2022-09-27T15:52:25.9056983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:25.9057988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:25.9059617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:25.9060565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:25.9313042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:25.9314007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:25.9315779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:25.9316362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:26.0569357Z ok (7.126s)
2022-09-27T15:52:26.0569614Z 
2022-09-27T15:52:26.0570017Z ----------------------------------------------------------------------
2022-09-27T15:52:26.0570344Z Ran 1 test in 7.126s
2022-09-27T15:52:26.0570523Z 
2022-09-27T15:52:26.0570623Z OK
2022-09-27T15:52:26.0570762Z 
2022-09-27T15:52:26.0570902Z Generating XML reports...
2022-09-27T15:52:26.0624731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155218.xml
2022-09-27T15:52:26.4602098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:26.4673811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:26.4674661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:26.4703369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:26.4711673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:26.5499111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:26.5499629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:26.5501638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:26.5502146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:26.5686455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:26.9909146Z ok (4.463s)
2022-09-27T15:52:26.9909376Z 
2022-09-27T15:52:26.9909782Z ----------------------------------------------------------------------
2022-09-27T15:52:26.9910122Z Ran 1 test in 4.463s
2022-09-27T15:52:26.9910279Z 
2022-09-27T15:52:26.9910390Z OK
2022-09-27T15:52:26.9910526Z 
2022-09-27T15:52:26.9910662Z Generating XML reports...
2022-09-27T15:52:26.9945542Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155222.xml
2022-09-27T15:52:27.0607221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:27.0629859Z 
2022-09-27T15:52:27.0630087Z Running tests...
2022-09-27T15:52:27.0630521Z ----------------------------------------------------------------------
2022-09-27T15:52:28.2615480Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:28.2616125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:28.2617065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:28.2617567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:28.7469056Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:28.7663976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11162
2022-09-27T15:52:28.7669510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11163
2022-09-27T15:52:28.7716683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:28.7741459Z 
2022-09-27T15:52:28.7741697Z Running tests...
2022-09-27T15:52:28.7742127Z ----------------------------------------------------------------------
2022-09-27T15:52:28.7761773Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:28.7762145Z 
2022-09-27T15:52:28.7762427Z ----------------------------------------------------------------------
2022-09-27T15:52:28.7762759Z Ran 1 test in 0.002s
2022-09-27T15:52:28.7762918Z 
2022-09-27T15:52:28.7763027Z OK (skipped=1)
2022-09-27T15:52:28.7763183Z 
2022-09-27T15:52:28.7763311Z Generating XML reports...
2022-09-27T15:52:28.7797177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155228.xml
2022-09-27T15:52:29.0224485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:29.0225044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:29.0226189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:29.0226758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:29.5713720Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:29.5736004Z 
2022-09-27T15:52:29.5736286Z Running tests...
2022-09-27T15:52:29.5737075Z ----------------------------------------------------------------------
2022-09-27T15:52:30.4117906Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:30.4118533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:30.4119148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:30.4119617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:30.4789909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:30.4790757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:30.4791786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:30.4792272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:30.7970742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:30.7971210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:30.7973552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:30.7974049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:31.0222286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:31.0222891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:31.0223691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:31.0224411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:31.0330690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:31.1236258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:31.1444733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:31.1445278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:31.1446071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:31.1447196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:31.1542493Z STAGE:2022-09-27 15:52:31 11162:11162 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:31.1543111Z STAGE:2022-09-27 15:52:31 11163:11163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:31.1559909Z STAGE:2022-09-27 15:52:31 11162:11162 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:31 11163:11163 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:31.1560589Z 
2022-09-27T15:52:31.1585741Z STAGE:2022-09-27 15:52:31 11162:11162 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:52:31 11163:11163 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:31.1586140Z 
2022-09-27T15:52:31.1600208Z STAGE:2022-09-27 15:52:31 11163:11163 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:31 11162:11162 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:31.1600828Z 
2022-09-27T15:52:31.2049760Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:31.2252440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11282
2022-09-27T15:52:31.2257476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11283
2022-09-27T15:52:31.2987289Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:31.3010199Z 
2022-09-27T15:52:31.3010468Z Running tests...
2022-09-27T15:52:31.3010892Z ----------------------------------------------------------------------
2022-09-27T15:52:31.3030077Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:31.3030423Z 
2022-09-27T15:52:31.3030691Z ----------------------------------------------------------------------
2022-09-27T15:52:31.3031044Z Ran 1 test in 0.002s
2022-09-27T15:52:31.3031197Z 
2022-09-27T15:52:31.3031308Z OK (skipped=1)
2022-09-27T15:52:31.3031466Z 
2022-09-27T15:52:31.3031593Z Generating XML reports...
2022-09-27T15:52:31.3067666Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155231.xml
2022-09-27T15:52:31.6746365Z ok (4.611s)
2022-09-27T15:52:31.6746585Z 
2022-09-27T15:52:31.6746967Z ----------------------------------------------------------------------
2022-09-27T15:52:31.6747291Z Ran 1 test in 4.612s
2022-09-27T15:52:31.6747458Z 
2022-09-27T15:52:31.6747552Z OK
2022-09-27T15:52:31.6747689Z 
2022-09-27T15:52:31.6747828Z Generating XML reports...
2022-09-27T15:52:31.6801304Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155227.xml
2022-09-27T15:52:33.0128243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:33.0129012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:33.0130630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:33.0131123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:33.0156428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:33.0156898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:33.0160507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:33.0161245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:33.3744149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:33.3745556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:33.3746836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:33.3747821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:33.5413124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:33.5471379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:33.5472941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:33.5514651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:33.5521420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:33.5524305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:33.6483597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:33.6486860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:33.6488261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:33.6538463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:33.6553117Z STAGE:2022-09-27 15:52:33 11282:11282 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:33.6596814Z STAGE:2022-09-27 15:52:33 11283:11283 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:33.8542087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:33.8543078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:33.8544229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:33.8545195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:33.8714818Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:33.8737294Z 
2022-09-27T15:52:33.8737888Z Running tests...
2022-09-27T15:52:33.8738395Z ----------------------------------------------------------------------
2022-09-27T15:52:33.8756765Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:33.8757144Z 
2022-09-27T15:52:33.8757507Z ----------------------------------------------------------------------
2022-09-27T15:52:33.8758034Z Ran 1 test in 0.002s
2022-09-27T15:52:33.8758222Z 
2022-09-27T15:52:33.8758333Z OK (skipped=1)
2022-09-27T15:52:33.8758491Z 
2022-09-27T15:52:33.8760912Z Generating XML reports...
2022-09-27T15:52:33.8792480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155233.xml
2022-09-27T15:52:34.3774873Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:34.3797333Z 
2022-09-27T15:52:34.3797587Z Running tests...
2022-09-27T15:52:34.3798040Z ----------------------------------------------------------------------
2022-09-27T15:52:35.3705432Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... [1664293955.370025] [28fd6574525b:11282:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:52:35.3706083Z [1664293955.370025] [28fd6574525b:11282:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:52:35.3812371Z [1664293955.380838] [28fd6574525b:11283:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:52:35.3812918Z [1664293955.380838] [28fd6574525b:11283:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:52:35.8162347Z STAGE:2022-09-27 15:52:35 11282:11282 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:35 11283:11283 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:35.8162792Z 
2022-09-27T15:52:35.8944178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:35.8944681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:35.8946746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:35.8947260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:35.9391970Z STAGE:2022-09-27 15:52:35 11282:11282 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:35.9395782Z STAGE:2022-09-27 15:52:35 11283:11283 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:35.9406804Z STAGE:2022-09-27 15:52:35 11282:11282 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:35.9407398Z STAGE:2022-09-27 15:52:35 11283:11283 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:36.0798951Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:36.1001354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11464
2022-09-27T15:52:36.1006630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11465
2022-09-27T15:52:36.4043373Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:36.4065838Z 
2022-09-27T15:52:36.4066099Z Running tests...
2022-09-27T15:52:36.4066543Z ----------------------------------------------------------------------
2022-09-27T15:52:36.4085561Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:36.4085932Z 
2022-09-27T15:52:36.4086211Z ----------------------------------------------------------------------
2022-09-27T15:52:36.4086547Z Ran 1 test in 0.002s
2022-09-27T15:52:36.4086710Z 
2022-09-27T15:52:36.4086802Z OK (skipped=1)
2022-09-27T15:52:36.4086957Z 
2022-09-27T15:52:36.4087081Z Generating XML reports...
2022-09-27T15:52:36.4119971Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155236.xml
2022-09-27T15:52:37.1399173Z ok (7.566s)
2022-09-27T15:52:37.1399443Z 
2022-09-27T15:52:37.1399840Z ----------------------------------------------------------------------
2022-09-27T15:52:37.1400513Z Ran 1 test in 7.566s
2022-09-27T15:52:37.1400724Z 
2022-09-27T15:52:37.1400844Z OK
2022-09-27T15:52:37.1400983Z 
2022-09-27T15:52:37.1401105Z Generating XML reports...
2022-09-27T15:52:37.1436564Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155229.xml
2022-09-27T15:52:37.7641031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:37.7641558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:37.7642384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:37.7642875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:37.8049724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:37.8050193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:37.8052902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:37.8053414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:38.3801746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:38.3802619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:38.3803194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:38.3803843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:38.3909007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:38.4185990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:38.4186481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:38.4190379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:38.4191128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:38.4817458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:38.5029970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:38.5030499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:38.5031291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:38.5032010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:38.5136028Z STAGE:2022-09-27 15:52:38 11464:11464 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:38.5193227Z STAGE:2022-09-27 15:52:38 11465:11465 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:38.5209310Z STAGE:2022-09-27 15:52:38 11465:11465 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:38.5211819Z STAGE:2022-09-27 15:52:38 11464:11464 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:38.5235882Z STAGE:2022-09-27 15:52:38 11465:11465 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:38.5247469Z STAGE:2022-09-27 15:52:38 11464:11464 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:38.5262031Z STAGE:2022-09-27 15:52:38 11465:11465 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:38.5264355Z STAGE:2022-09-27 15:52:38 11464:11464 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:38.9180081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:38.9201986Z 
2022-09-27T15:52:38.9202431Z Running tests...
2022-09-27T15:52:38.9202901Z ----------------------------------------------------------------------
2022-09-27T15:52:38.9223057Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:38.9223414Z 
2022-09-27T15:52:38.9223708Z ----------------------------------------------------------------------
2022-09-27T15:52:38.9224041Z Ran 1 test in 0.002s
2022-09-27T15:52:38.9224205Z 
2022-09-27T15:52:38.9224297Z OK (skipped=1)
2022-09-27T15:52:38.9224455Z 
2022-09-27T15:52:38.9224582Z Generating XML reports...
2022-09-27T15:52:38.9258152Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155238.xml
2022-09-27T15:52:39.0077965Z ok (4.628s)
2022-09-27T15:52:39.0078391Z 
2022-09-27T15:52:39.0079038Z ----------------------------------------------------------------------
2022-09-27T15:52:39.0079723Z Ran 1 test in 4.628s
2022-09-27T15:52:39.0080071Z 
2022-09-27T15:52:39.0080576Z OK
2022-09-27T15:52:39.0080862Z 
2022-09-27T15:52:39.0081109Z Generating XML reports...
2022-09-27T15:52:39.0114021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155234.xml
2022-09-27T15:52:39.2815399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:39.2815897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:39.2816813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:39.2817298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:39.7677917Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:39.7700107Z 
2022-09-27T15:52:39.7700354Z Running tests...
2022-09-27T15:52:39.7700799Z ----------------------------------------------------------------------
2022-09-27T15:52:40.8768422Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:40.8769034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:40.8770272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:40.8770764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:41.1635396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:41.1635904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:41.1637331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:41.1637826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:41.3858179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:41.3882341Z 
2022-09-27T15:52:41.3882587Z Running tests...
2022-09-27T15:52:41.3883035Z ----------------------------------------------------------------------
2022-09-27T15:52:41.3902398Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:41.3902742Z 
2022-09-27T15:52:41.3903010Z ----------------------------------------------------------------------
2022-09-27T15:52:41.3903324Z Ran 1 test in 0.002s
2022-09-27T15:52:41.3903493Z 
2022-09-27T15:52:41.3903604Z OK (skipped=1)
2022-09-27T15:52:41.3903763Z 
2022-09-27T15:52:41.3903910Z Generating XML reports...
2022-09-27T15:52:41.3940094Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155241.xml
2022-09-27T15:52:41.4400042Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:41.4603731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11682
2022-09-27T15:52:41.4609123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11683
2022-09-27T15:52:41.6685794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:41.6708594Z 
2022-09-27T15:52:41.6708958Z Running tests...
2022-09-27T15:52:41.6709435Z ----------------------------------------------------------------------
2022-09-27T15:52:43.1457160Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:43.1458730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:43.1459934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:43.1460861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:43.1802399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:43.1803371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:43.1807101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:43.1808062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:43.3571485Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:43.3769876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11782
2022-09-27T15:52:43.3775922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11783
2022-09-27T15:52:43.3793100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:43.3794352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:43.3799002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:43.3799985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:43.6568132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:43.7022862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:43.7024338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:43.7074968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:43.7082051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:43.7085315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:43.8036455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:43.8038359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:43.8039817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:43.8100100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:43.8106567Z STAGE:2022-09-27 15:52:43 11682:11682 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:43.8158742Z STAGE:2022-09-27 15:52:43 11683:11683 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:43.8729283Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:43.8752919Z 
2022-09-27T15:52:43.8753402Z Running tests...
2022-09-27T15:52:43.8754261Z ----------------------------------------------------------------------
2022-09-27T15:52:43.8773844Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:43.8774529Z 
2022-09-27T15:52:43.8775101Z ----------------------------------------------------------------------
2022-09-27T15:52:43.8775495Z Ran 1 test in 0.002s
2022-09-27T15:52:43.8775683Z 
2022-09-27T15:52:43.8775792Z OK (skipped=1)
2022-09-27T15:52:43.8775956Z 
2022-09-27T15:52:43.8776092Z Generating XML reports...
2022-09-27T15:52:43.8811233Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155243.xml
2022-09-27T15:52:45.0563341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:45.0564358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:45.0565505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:45.0566436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:45.0688064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:45.0688936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:45.0690482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:45.0691351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:45.5422938Z [1664293965.541721] [28fd6574525b:11683:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:52:45.5424319Z [1664293965.541721] [28fd6574525b:11683:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:52:45.5445947Z [1664293965.544249] [28fd6574525b:11682:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:52:45.5446943Z [1664293965.544249] [28fd6574525b:11682:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:52:45.6113682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:45.6114734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:45.6115968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:45.6116694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:45.6221859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:45.7129146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:45.7236548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:45.7237480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:45.7238896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:45.7240577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:45.7306392Z STAGE:2022-09-27 15:52:45 11782:11782 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:45.7399036Z STAGE:2022-09-27 15:52:45 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:45.7418267Z STAGE:2022-09-27 15:52:45 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:45.7419407Z STAGE:2022-09-27 15:52:45 11782:11782 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:45.7447958Z STAGE:2022-09-27 15:52:45 11783:11783 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:45.7448606Z STAGE:2022-09-27 15:52:45 11782:11782 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:45.7464819Z STAGE:2022-09-27 15:52:45 11783:11783 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:45 11782:11782 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:45.7465931Z 
2022-09-27T15:52:45.9030430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:45.9031673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:45.9033213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:45.9034444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:45.9291842Z STAGE:2022-09-27 15:52:45 11683:11683 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:45.9309888Z STAGE:2022-09-27 15:52:45 11682:11682 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:46.0540350Z STAGE:2022-09-27 15:52:46 11683:11683 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:46.0578494Z STAGE:2022-09-27 15:52:46 11682:11682 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:46.0589190Z STAGE:2022-09-27 15:52:46 11683:11683 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:46.0590386Z STAGE:2022-09-27 15:52:46 11682:11682 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:46.1843755Z ok (4.513s)
2022-09-27T15:52:46.1843986Z 
2022-09-27T15:52:46.1844389Z ----------------------------------------------------------------------
2022-09-27T15:52:46.1844716Z Ran 1 test in 4.513s
2022-09-27T15:52:46.1844891Z 
2022-09-27T15:52:46.1844987Z OK
2022-09-27T15:52:46.1845126Z 
2022-09-27T15:52:46.1845261Z Generating XML reports...
2022-09-27T15:52:46.1880005Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155241.xml
2022-09-27T15:52:46.4235830Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:46.4258378Z 
2022-09-27T15:52:46.4258647Z Running tests...
2022-09-27T15:52:46.4259120Z ----------------------------------------------------------------------
2022-09-27T15:52:46.4278047Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:46.4278415Z 
2022-09-27T15:52:46.4278667Z ----------------------------------------------------------------------
2022-09-27T15:52:46.4279004Z Ran 1 test in 0.002s
2022-09-27T15:52:46.4279175Z 
2022-09-27T15:52:46.4279285Z OK (skipped=1)
2022-09-27T15:52:46.4279445Z 
2022-09-27T15:52:46.4279553Z Generating XML reports...
2022-09-27T15:52:46.4313399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155246.xml
2022-09-27T15:52:47.2744320Z ok (7.504s)
2022-09-27T15:52:47.2745489Z 
2022-09-27T15:52:47.2746965Z ----------------------------------------------------------------------
2022-09-27T15:52:47.2747956Z Ran 1 test in 7.504s
2022-09-27T15:52:47.2748676Z 
2022-09-27T15:52:47.2749323Z OK
2022-09-27T15:52:47.2750047Z 
2022-09-27T15:52:47.2750732Z Generating XML reports...
2022-09-27T15:52:47.2811023Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155239.xml
2022-09-27T15:52:48.3934257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:48.3935097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:48.3936729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:48.3937233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:48.4326689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:48.4327885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:48.4330101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:48.4331313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:48.8894125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:48.8917661Z 
2022-09-27T15:52:48.8918013Z Running tests...
2022-09-27T15:52:48.8918445Z ----------------------------------------------------------------------
2022-09-27T15:52:48.9300908Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:48.9325151Z 
2022-09-27T15:52:48.9325372Z Running tests...
2022-09-27T15:52:48.9325807Z ----------------------------------------------------------------------
2022-09-27T15:52:48.9346264Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:48.9346963Z 
2022-09-27T15:52:48.9347466Z ----------------------------------------------------------------------
2022-09-27T15:52:48.9348127Z Ran 1 test in 0.002s
2022-09-27T15:52:48.9348435Z 
2022-09-27T15:52:48.9349148Z OK (skipped=1)
2022-09-27T15:52:48.9349442Z 
2022-09-27T15:52:48.9349689Z Generating XML reports...
2022-09-27T15:52:48.9382537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155248.xml
2022-09-27T15:52:49.3897680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:49.3898687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:49.3900511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:49.3901507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:49.8932954Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:52:49.8970795Z 
2022-09-27T15:52:49.8971252Z Running tests...
2022-09-27T15:52:49.8972319Z ----------------------------------------------------------------------
2022-09-27T15:52:50.5743931Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:50.5949691Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12049
2022-09-27T15:52:50.5954731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12050
2022-09-27T15:52:50.8705677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:50.8706191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:50.8709235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:50.8709732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:51.4805508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:51.4828067Z 
2022-09-27T15:52:51.4828580Z Running tests...
2022-09-27T15:52:51.4829080Z ----------------------------------------------------------------------
2022-09-27T15:52:51.4847395Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:52:51.4848238Z 
2022-09-27T15:52:51.4848558Z ----------------------------------------------------------------------
2022-09-27T15:52:51.4848913Z Ran 1 test in 0.002s
2022-09-27T15:52:51.4849089Z 
2022-09-27T15:52:51.4849204Z OK (skipped=1)
2022-09-27T15:52:51.4849365Z 
2022-09-27T15:52:51.4849493Z Generating XML reports...
2022-09-27T15:52:51.4886274Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155251.xml
2022-09-27T15:52:51.5167163Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:51.5368228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12115
2022-09-27T15:52:51.5372872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12116
2022-09-27T15:52:52.2819291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:52.2819814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:52.2820609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:52.2821081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:52.3434444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:52.3434940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:52.3437401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:52.3437879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:52.9378882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:52.9379456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:52.9380257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:52.9380950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:52.9386672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:52.9387191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:53.2068900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:53.2069446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:53.2070468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:53.2070999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:53.2798437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:53.2798945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:53.2802196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:53.2802685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:53.3020649Z skip: Skipped due to small world size. (4.410s)
2022-09-27T15:52:53.3021094Z 
2022-09-27T15:52:53.3021681Z ----------------------------------------------------------------------
2022-09-27T15:52:53.3022292Z Ran 1 test in 4.410s
2022-09-27T15:52:53.3022611Z 
2022-09-27T15:52:53.3022824Z OK (skipped=1)
2022-09-27T15:52:53.3023114Z 
2022-09-27T15:52:53.3023353Z Generating XML reports...
2022-09-27T15:52:53.3061269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155248.xml
2022-09-27T15:52:53.4684206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:53.4685102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:53.4687051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:53.4688358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:53.7166951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:53.8025119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:53.8025950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:53.8078721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:53.8086488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:53.8089787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:52:53.9037707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:52:53.9040285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:52:53.9041395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:53.9103930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:52:53.9107122Z STAGE:2022-09-27 15:52:53 12116:12116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:53.9163117Z STAGE:2022-09-27 15:52:53 12115:12115 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:53.9587174Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:53.9610341Z 
2022-09-27T15:52:53.9610857Z Running tests...
2022-09-27T15:52:53.9611349Z ----------------------------------------------------------------------
2022-09-27T15:52:53.9629583Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:53.9630427Z 
2022-09-27T15:52:53.9630712Z ----------------------------------------------------------------------
2022-09-27T15:52:53.9631076Z Ran 1 test in 0.002s
2022-09-27T15:52:53.9631225Z 
2022-09-27T15:52:53.9631347Z OK (skipped=1)
2022-09-27T15:52:53.9631506Z 
2022-09-27T15:52:53.9631633Z Generating XML reports...
2022-09-27T15:52:53.9667502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155253.xml
2022-09-27T15:52:55.4907412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:55.4908408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:55.4910153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:55.4911124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:55.6201734Z [1664293975.619440] [28fd6574525b:12116:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:52:55.6202903Z [1664293975.619440] [28fd6574525b:12116:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:52:55.6378482Z [1664293975.637401] [28fd6574525b:12115:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:52:55.6379583Z [1664293975.637401] [28fd6574525b:12115:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:52:56.0004868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:52:56.0027753Z 
2022-09-27T15:52:56.0028320Z Running tests...
2022-09-27T15:52:56.0028832Z ----------------------------------------------------------------------
2022-09-27T15:52:56.0184554Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 15:52:56 12116:12116 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:52:56 12115:12115 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:56.0185160Z 
2022-09-27T15:52:56.0355532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:56.0355999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:56.0358801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:56.0359294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:56.1451559Z STAGE:2022-09-27 15:52:56 12115:12115 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:56.1478637Z STAGE:2022-09-27 15:52:56 12116:12116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:52:56.1490469Z STAGE:2022-09-27 15:52:56 12115:12115 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:56.1491622Z STAGE:2022-09-27 15:52:56 12116:12116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:52:56.5398554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:56.5421015Z 
2022-09-27T15:52:56.5421495Z Running tests...
2022-09-27T15:52:56.5421982Z ----------------------------------------------------------------------
2022-09-27T15:52:56.5438986Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:56.5440639Z 
2022-09-27T15:52:56.5441106Z ----------------------------------------------------------------------
2022-09-27T15:52:56.5441453Z Ran 1 test in 0.002s
2022-09-27T15:52:56.5441622Z 
2022-09-27T15:52:56.5441747Z OK (skipped=1)
2022-09-27T15:52:56.5441905Z 
2022-09-27T15:52:56.5442052Z Generating XML reports...
2022-09-27T15:52:56.5475016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155256.xml
2022-09-27T15:52:57.3498172Z ok (7.453s)
2022-09-27T15:52:57.3498386Z 
2022-09-27T15:52:57.3498783Z ----------------------------------------------------------------------
2022-09-27T15:52:57.3499145Z Ran 1 test in 7.453s
2022-09-27T15:52:57.3499297Z 
2022-09-27T15:52:57.3499397Z OK
2022-09-27T15:52:57.3499534Z 
2022-09-27T15:52:57.3499671Z Generating XML reports...
2022-09-27T15:52:57.3534608Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155249.xml
2022-09-27T15:52:57.7258344Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:52:57.7459729Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12339
2022-09-27T15:52:57.7465723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12340
2022-09-27T15:52:58.5491749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:58.5496402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:58.5515433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:58.5517461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:59.0961524Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:52:59.0983778Z 
2022-09-27T15:52:59.0983931Z Running tests...
2022-09-27T15:52:59.0984963Z ----------------------------------------------------------------------
2022-09-27T15:52:59.1002859Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T15:52:59.1003199Z 
2022-09-27T15:52:59.1003474Z ----------------------------------------------------------------------
2022-09-27T15:52:59.1004073Z Ran 1 test in 0.002s
2022-09-27T15:52:59.1004258Z 
2022-09-27T15:52:59.1004374Z OK (skipped=1)
2022-09-27T15:52:59.1004536Z 
2022-09-27T15:52:59.1004667Z Generating XML reports...
2022-09-27T15:52:59.1037829Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155259.xml
2022-09-27T15:52:59.4015629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:59.4016165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:59.4017939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:59.4018421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:59.4344075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:59.4344584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:59.4347575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:59.4348457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:59.5703671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:52:59.5704172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:52:59.5706434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:52:59.5706947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:52:59.9681402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:52:59.9682009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:52:59.9682829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:59.9683549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:52:59.9690248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:52:59.9690762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:00.1221317Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:53:00.1244769Z 
2022-09-27T15:53:00.1245146Z Running tests...
2022-09-27T15:53:00.1245587Z ----------------------------------------------------------------------
2022-09-27T15:53:00.4545000Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Skipped due to small world size. (4.451s)
2022-09-27T15:53:00.4545374Z 
2022-09-27T15:53:00.4545769Z ----------------------------------------------------------------------
2022-09-27T15:53:00.4546112Z Ran 1 test in 4.452s
2022-09-27T15:53:00.4546293Z 
2022-09-27T15:53:00.4546385Z OK (skipped=1)
2022-09-27T15:53:00.4546544Z 
2022-09-27T15:53:00.4546673Z Generating XML reports...
2022-09-27T15:53:00.4602370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155255.xml
2022-09-27T15:53:01.1090404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:01.1090926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:01.1092698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:01.1093211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:01.6844070Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:01.6888525Z 
2022-09-27T15:53:01.6889309Z Running tests...
2022-09-27T15:53:01.6890405Z ----------------------------------------------------------------------
2022-09-27T15:53:01.6933895Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.004s)
2022-09-27T15:53:01.6943815Z 
2022-09-27T15:53:01.6949220Z ----------------------------------------------------------------------
2022-09-27T15:53:01.6950064Z Ran 1 test in 0.005s
2022-09-27T15:53:01.6950416Z 
2022-09-27T15:53:01.6950657Z OK (skipped=1)
2022-09-27T15:53:01.6951005Z 
2022-09-27T15:53:01.6951279Z Generating XML reports...
2022-09-27T15:53:01.7011485Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155301.xml
2022-09-27T15:53:01.7807384Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:01.8006334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12514
2022-09-27T15:53:01.8011158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12515
2022-09-27T15:53:02.6881635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:02.6882156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:02.6884478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:02.6884981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:03.2362704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:03.2384768Z 
2022-09-27T15:53:03.2385095Z Running tests...
2022-09-27T15:53:03.2385525Z ----------------------------------------------------------------------
2022-09-27T15:53:03.4820820Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:03.4821807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:03.4822630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:03.4823125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:03.5070977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:03.5071448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:03.5074106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:03.5074598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:03.7555890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:03.7556694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:03.7557774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:03.7558268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:03.9923204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:04.0236028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:04.0329088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:04.0329990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:04.0336680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:04.0339754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:53:04.1249226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:04.1251511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:53:04.1252722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:53:04.1253473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:53:04.1348974Z STAGE:2022-09-27 15:53:04 12515:12515 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:04.1349581Z STAGE:2022-09-27 15:53:04 12514:12514 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:04.2585118Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:04.2607972Z 
2022-09-27T15:53:04.2608537Z Running tests...
2022-09-27T15:53:04.2609021Z ----------------------------------------------------------------------
2022-09-27T15:53:04.2627156Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:04.2627484Z 
2022-09-27T15:53:04.2627984Z ----------------------------------------------------------------------
2022-09-27T15:53:04.2628377Z Ran 1 test in 0.002s
2022-09-27T15:53:04.2628556Z 
2022-09-27T15:53:04.2628665Z OK (skipped=1)
2022-09-27T15:53:04.2628858Z 
2022-09-27T15:53:04.2629001Z Generating XML reports...
2022-09-27T15:53:04.2662996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155304.xml
2022-09-27T15:53:04.8384816Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:04.8586628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12627
2022-09-27T15:53:04.8591464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12628
2022-09-27T15:53:05.8817715Z [1664293985.881161] [28fd6574525b:12515:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:53:05.8818773Z [1664293985.881161] [28fd6574525b:12515:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:53:05.8946526Z [1664293985.894151] [28fd6574525b:12514:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:53:05.8947532Z [1664293985.894151] [28fd6574525b:12514:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:53:06.2785681Z STAGE:2022-09-27 15:53:06 12515:12515 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:06 12514:12514 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:06.2786133Z 
2022-09-27T15:53:06.4014658Z STAGE:2022-09-27 15:53:06 12515:12515 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:06.4067579Z STAGE:2022-09-27 15:53:06 12514:12514 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:06.4077828Z STAGE:2022-09-27 15:53:06 12515:12515 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:06.4078414Z STAGE:2022-09-27 15:53:06 12514:12514 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:06.4101405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:06.4101881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:06.4105840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:06.4106341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:06.6260310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:06.6260828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:06.6263700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:06.6264176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:06.6378759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:06.6379235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:06.6381722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:06.6382232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:06.9250173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:06.9272326Z 
2022-09-27T15:53:06.9272609Z Running tests...
2022-09-27T15:53:06.9273040Z ----------------------------------------------------------------------
2022-09-27T15:53:06.9292173Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:06.9292576Z 
2022-09-27T15:53:06.9293044Z ----------------------------------------------------------------------
2022-09-27T15:53:06.9293377Z Ran 1 test in 0.002s
2022-09-27T15:53:06.9293528Z 
2022-09-27T15:53:06.9293637Z OK (skipped=1)
2022-09-27T15:53:06.9293791Z 
2022-09-27T15:53:06.9293916Z Generating XML reports...
2022-09-27T15:53:06.9328090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155306.xml
2022-09-27T15:53:07.1656345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:07.1656866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:07.1657655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:07.1658352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:07.1764824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:07.2672413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:07.6136641Z ok (7.489s)
2022-09-27T15:53:07.6136856Z 
2022-09-27T15:53:07.6137245Z ----------------------------------------------------------------------
2022-09-27T15:53:07.6137700Z Ran 1 test in 7.489s
2022-09-27T15:53:07.6138033Z 
2022-09-27T15:53:07.6138161Z OK
2022-09-27T15:53:07.6138285Z 
2022-09-27T15:53:07.6138422Z Generating XML reports...
2022-09-27T15:53:07.6174538Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155300.xml
2022-09-27T15:53:07.6659271Z skip: Skipped due to small world size. (4.427s)
2022-09-27T15:53:07.6659519Z 
2022-09-27T15:53:07.6659895Z ----------------------------------------------------------------------
2022-09-27T15:53:07.6660220Z Ran 1 test in 4.427s
2022-09-27T15:53:07.6660387Z 
2022-09-27T15:53:07.6660496Z OK (skipped=1)
2022-09-27T15:53:07.6660656Z 
2022-09-27T15:53:07.6660783Z Generating XML reports...
2022-09-27T15:53:07.6696386Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155303.xml
2022-09-27T15:53:08.9711598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:08.9712457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:08.9714715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:08.9715504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:09.4590418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:09.4613696Z 
2022-09-27T15:53:09.4614176Z Running tests...
2022-09-27T15:53:09.4614642Z ----------------------------------------------------------------------
2022-09-27T15:53:09.4634573Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:09.4635059Z 
2022-09-27T15:53:09.4635726Z ----------------------------------------------------------------------
2022-09-27T15:53:09.4636098Z Ran 1 test in 0.002s
2022-09-27T15:53:09.4636267Z 
2022-09-27T15:53:09.4636391Z OK (skipped=1)
2022-09-27T15:53:09.4636552Z 
2022-09-27T15:53:09.4636680Z Generating XML reports...
2022-09-27T15:53:09.4671805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155309.xml
2022-09-27T15:53:09.8253460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:09.8254155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:09.8255495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:09.8255988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:09.8623211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:09.8623674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:09.8626578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:09.8627077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:10.3134478Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:53:10.3156451Z 
2022-09-27T15:53:10.3156738Z Running tests...
2022-09-27T15:53:10.3157189Z ----------------------------------------------------------------------
2022-09-27T15:53:10.3555018Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:10.3597962Z 
2022-09-27T15:53:10.3598279Z Running tests...
2022-09-27T15:53:10.3599250Z ----------------------------------------------------------------------
2022-09-27T15:53:11.3771126Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:11.3771769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:11.3772691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:11.3773204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:11.8993373Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:11.9016565Z 
2022-09-27T15:53:11.9016870Z Running tests...
2022-09-27T15:53:11.9017321Z ----------------------------------------------------------------------
2022-09-27T15:53:11.9036082Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:11.9036430Z 
2022-09-27T15:53:11.9036942Z ----------------------------------------------------------------------
2022-09-27T15:53:11.9037316Z Ran 1 test in 0.002s
2022-09-27T15:53:11.9037752Z 
2022-09-27T15:53:11.9037888Z OK (skipped=1)
2022-09-27T15:53:11.9038049Z 
2022-09-27T15:53:11.9038177Z Generating XML reports...
2022-09-27T15:53:11.9072438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155311.xml
2022-09-27T15:53:11.9262869Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:11.9467052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12873
2022-09-27T15:53:11.9472454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12874
2022-09-27T15:53:12.0069793Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:12.0277243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12876
2022-09-27T15:53:12.0282672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12877
2022-09-27T15:53:13.6243182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:13.6243759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:13.6244372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:13.6245278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:13.6475187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:13.6475980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:13.6477605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:13.6478076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:13.7129265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:13.7130155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:13.7131377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:13.7131858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:13.7838017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:13.7838813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:13.7839907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:13.7840669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:13.9958781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:13.9959411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:13.9962002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:13.9962484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:14.1433044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:14.1564502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:14.1565283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:14.1635926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:14.1642213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:14.2579086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:14.3182719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:14.3183311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:14.3184102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:14.3184807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:14.3292642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:14.4196414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:14.5081539Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:14.5105343Z 
2022-09-27T15:53:14.5105886Z Running tests...
2022-09-27T15:53:14.5106399Z ----------------------------------------------------------------------
2022-09-27T15:53:14.5126031Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:14.5126655Z 
2022-09-27T15:53:14.5126944Z ----------------------------------------------------------------------
2022-09-27T15:53:14.5127262Z Ran 1 test in 0.002s
2022-09-27T15:53:14.5127429Z 
2022-09-27T15:53:14.5127538Z OK (skipped=1)
2022-09-27T15:53:14.5127694Z 
2022-09-27T15:53:14.5127819Z Generating XML reports...
2022-09-27T15:53:14.5163542Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155314.xml
2022-09-27T15:53:14.6545754Z skip: Skipped due to small world size. (4.338s)
2022-09-27T15:53:14.6545990Z 
2022-09-27T15:53:14.6546359Z ----------------------------------------------------------------------
2022-09-27T15:53:14.6546702Z Ran 1 test in 4.339s
2022-09-27T15:53:14.6546885Z 
2022-09-27T15:53:14.6546996Z OK (skipped=1)
2022-09-27T15:53:14.6547157Z 
2022-09-27T15:53:14.6547266Z Generating XML reports...
2022-09-27T15:53:14.6583143Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155310.xml
2022-09-27T15:53:14.8352252Z skip: Skipped due to small world size. (4.475s)
2022-09-27T15:53:14.8352508Z 
2022-09-27T15:53:14.8352872Z ----------------------------------------------------------------------
2022-09-27T15:53:14.8353194Z Ran 1 test in 4.475s
2022-09-27T15:53:14.8353362Z 
2022-09-27T15:53:14.8353470Z OK (skipped=1)
2022-09-27T15:53:14.8353627Z 
2022-09-27T15:53:14.8353753Z Generating XML reports...
2022-09-27T15:53:14.8389509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155310.xml
2022-09-27T15:53:16.6333218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:16.6333812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:16.6335687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:16.6336200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:16.8497308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:16.8497821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:16.8500131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:16.8500652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:17.0550047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:17.0550889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:17.0552069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:17.0552585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:17.1372969Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:17.1397173Z 
2022-09-27T15:53:17.1397602Z Running tests...
2022-09-27T15:53:17.1398386Z ----------------------------------------------------------------------
2022-09-27T15:53:17.1417707Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:17.1418302Z 
2022-09-27T15:53:17.1418628Z ----------------------------------------------------------------------
2022-09-27T15:53:17.1418968Z Ran 1 test in 0.002s
2022-09-27T15:53:17.1419141Z 
2022-09-27T15:53:17.1419252Z OK (skipped=1)
2022-09-27T15:53:17.1419394Z 
2022-09-27T15:53:17.1419535Z Generating XML reports...
2022-09-27T15:53:17.1454532Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155317.xml
2022-09-27T15:53:17.3421053Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:53:17.3443031Z 
2022-09-27T15:53:17.3443468Z Running tests...
2022-09-27T15:53:17.3444011Z ----------------------------------------------------------------------
2022-09-27T15:53:17.5486581Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:17.5508410Z 
2022-09-27T15:53:17.5508865Z Running tests...
2022-09-27T15:53:17.5509357Z ----------------------------------------------------------------------
2022-09-27T15:53:18.9542076Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:18.9733351Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13183
2022-09-27T15:53:18.9739486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13184
2022-09-27T15:53:19.0925296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:19.0926290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:19.0927452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:19.0928396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:19.1021083Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:19.1217550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13187
2022-09-27T15:53:19.1223517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13188
2022-09-27T15:53:19.5947079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:19.5987112Z 
2022-09-27T15:53:19.5987652Z Running tests...
2022-09-27T15:53:19.5988555Z ----------------------------------------------------------------------
2022-09-27T15:53:19.6024246Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.004s)
2022-09-27T15:53:19.6025219Z 
2022-09-27T15:53:19.6028521Z ----------------------------------------------------------------------
2022-09-27T15:53:19.6031216Z Ran 1 test in 0.004s
2022-09-27T15:53:19.6034328Z 
2022-09-27T15:53:19.6037853Z OK (skipped=1)
2022-09-27T15:53:19.6041018Z 
2022-09-27T15:53:19.6045120Z Generating XML reports...
2022-09-27T15:53:19.6107363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155319.xml
2022-09-27T15:53:20.7863842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:20.7871459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:20.7874928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:20.7877959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:20.8753753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:20.8754253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:20.8756458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:20.8756956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:20.9116696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:20.9117205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:20.9119316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:20.9120081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:21.0136986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:21.0137478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:21.0139198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:21.0139694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:21.3221411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:21.3906242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:21.3907084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:21.3931154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:21.3937991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:21.4918833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:21.5555369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:21.5555920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:21.5556703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:21.5557407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:21.5665590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:21.5735435Z STAGE:2022-09-27 15:53:21 13188:13188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:21.6571143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:21.6639422Z STAGE:2022-09-27 15:53:21 13187:13187 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:21.6656818Z STAGE:2022-09-27 15:53:21 13187:13187 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:21.6657876Z STAGE:2022-09-27 15:53:21 13188:13188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:21.6683172Z STAGE:2022-09-27 15:53:21 13187:13187 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:21.6684681Z STAGE:2022-09-27 15:53:21 13188:13188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:21.6698080Z STAGE:2022-09-27 15:53:21 13187:13187 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:21 13188:13188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:21.6698737Z 
2022-09-27T15:53:21.7381712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:21.7382235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:21.7383712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:21.7384209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:21.8808914Z skip: Skipped due to small world size. (4.536s)
2022-09-27T15:53:21.8809176Z 
2022-09-27T15:53:21.8809570Z ----------------------------------------------------------------------
2022-09-27T15:53:21.8809918Z Ran 1 test in 4.536s
2022-09-27T15:53:21.8810088Z 
2022-09-27T15:53:21.8810195Z OK (skipped=1)
2022-09-27T15:53:21.8810341Z 
2022-09-27T15:53:21.8810728Z Generating XML reports...
2022-09-27T15:53:21.8845731Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155317.xml
2022-09-27T15:53:22.0303943Z ok (4.479s)
2022-09-27T15:53:22.0304155Z 
2022-09-27T15:53:22.0304526Z ----------------------------------------------------------------------
2022-09-27T15:53:22.0304851Z Ran 1 test in 4.479s
2022-09-27T15:53:22.0305026Z 
2022-09-27T15:53:22.0305123Z OK
2022-09-27T15:53:22.0305257Z 
2022-09-27T15:53:22.0305393Z Generating XML reports...
2022-09-27T15:53:22.0360871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155317.xml
2022-09-27T15:53:22.2906269Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:22.2928933Z 
2022-09-27T15:53:22.2929289Z Running tests...
2022-09-27T15:53:22.2929776Z ----------------------------------------------------------------------
2022-09-27T15:53:22.2949890Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:22.2950211Z 
2022-09-27T15:53:22.2950495Z ----------------------------------------------------------------------
2022-09-27T15:53:22.2950826Z Ran 1 test in 0.002s
2022-09-27T15:53:22.2950991Z 
2022-09-27T15:53:22.2951106Z OK (skipped=1)
2022-09-27T15:53:22.2951263Z 
2022-09-27T15:53:22.2951373Z Generating XML reports...
2022-09-27T15:53:22.2986195Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155322.xml
2022-09-27T15:53:24.0694162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:24.0694710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:24.0695883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:24.0696371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:24.2698926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:24.2699451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:24.2701455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:24.2701938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:24.3370568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:24.3371051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:24.3374749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:24.3375253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:24.5520930Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:53:24.5543188Z 
2022-09-27T15:53:24.5543549Z Running tests...
2022-09-27T15:53:24.5544013Z ----------------------------------------------------------------------
2022-09-27T15:53:24.7738728Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:24.7761884Z 
2022-09-27T15:53:24.7762090Z Running tests...
2022-09-27T15:53:24.7762512Z ----------------------------------------------------------------------
2022-09-27T15:53:24.8406121Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:24.8429819Z 
2022-09-27T15:53:24.8430180Z Running tests...
2022-09-27T15:53:24.8430969Z ----------------------------------------------------------------------
2022-09-27T15:53:24.8450675Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:24.8450997Z 
2022-09-27T15:53:24.8451293Z ----------------------------------------------------------------------
2022-09-27T15:53:24.8451612Z Ran 1 test in 0.002s
2022-09-27T15:53:24.8451783Z 
2022-09-27T15:53:24.8451892Z OK (skipped=1)
2022-09-27T15:53:24.8452048Z 
2022-09-27T15:53:24.8452176Z Generating XML reports...
2022-09-27T15:53:24.8488315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155324.xml
2022-09-27T15:53:26.1346813Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:26.1542870Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13498
2022-09-27T15:53:26.1548352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13499
2022-09-27T15:53:26.4083807Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:26.4286143Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13501
2022-09-27T15:53:26.4292199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13502
2022-09-27T15:53:26.7539346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:26.7539865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:26.7542032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:26.7542525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:27.4010183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:27.4031832Z 
2022-09-27T15:53:27.4032105Z Running tests...
2022-09-27T15:53:27.4032564Z ----------------------------------------------------------------------
2022-09-27T15:53:27.4052515Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:27.4053069Z 
2022-09-27T15:53:27.4053500Z ----------------------------------------------------------------------
2022-09-27T15:53:27.4053848Z Ran 1 test in 0.002s
2022-09-27T15:53:27.4054019Z 
2022-09-27T15:53:27.4054126Z OK (skipped=1)
2022-09-27T15:53:27.4054285Z 
2022-09-27T15:53:27.4055831Z Generating XML reports...
2022-09-27T15:53:27.4088913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155327.xml
2022-09-27T15:53:27.8115050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:27.8115914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:27.8117124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:27.8117630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:27.8197484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:27.8197966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:27.8201369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:27.8201863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:28.0710580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:28.0711154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:28.0713519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:28.0714293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:28.1378382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:28.1378872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:28.1381319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:28.1381798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:28.3353017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:28.3398860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:28.3403219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:28.3454737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:28.3504810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:28.4410522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:28.7268039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:28.7268620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:28.7269422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:28.7270131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:28.7376271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:28.7610833Z skip: Skipped due to small world size. (4.206s)
2022-09-27T15:53:28.7611254Z 
2022-09-27T15:53:28.7611645Z ----------------------------------------------------------------------
2022-09-27T15:53:28.7611974Z Ran 1 test in 4.207s
2022-09-27T15:53:28.7612140Z 
2022-09-27T15:53:28.7612254Z OK (skipped=1)
2022-09-27T15:53:28.7612411Z 
2022-09-27T15:53:28.7612544Z Generating XML reports...
2022-09-27T15:53:28.7649104Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155324.xml
2022-09-27T15:53:28.8282316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:29.2360667Z ok (4.459s)
2022-09-27T15:53:29.2361025Z 
2022-09-27T15:53:29.2361727Z ----------------------------------------------------------------------
2022-09-27T15:53:29.2362103Z Ran 1 test in 4.460s
2022-09-27T15:53:29.2362273Z 
2022-09-27T15:53:29.2362371Z OK
2022-09-27T15:53:29.2362510Z 
2022-09-27T15:53:29.2362640Z Generating XML reports...
2022-09-27T15:53:29.2399134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155324.xml
2022-09-27T15:53:29.3822752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:29.3823251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:29.3825450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:29.3825960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:29.9079886Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:29.9121428Z 
2022-09-27T15:53:29.9123499Z Running tests...
2022-09-27T15:53:29.9126225Z ----------------------------------------------------------------------
2022-09-27T15:53:29.9162512Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.004s)
2022-09-27T15:53:29.9165516Z 
2022-09-27T15:53:29.9167902Z ----------------------------------------------------------------------
2022-09-27T15:53:29.9170467Z Ran 1 test in 0.004s
2022-09-27T15:53:29.9172577Z 
2022-09-27T15:53:29.9174551Z OK (skipped=1)
2022-09-27T15:53:29.9176653Z 
2022-09-27T15:53:29.9178791Z Generating XML reports...
2022-09-27T15:53:29.9227487Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155329.xml
2022-09-27T15:53:31.0255967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:31.0256509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:31.0261082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:31.0261590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:31.3759131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:31.3759655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:31.3762022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:31.3762529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:31.5320628Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:53:31.5344360Z 
2022-09-27T15:53:31.5344783Z Running tests...
2022-09-27T15:53:31.5345322Z ----------------------------------------------------------------------
2022-09-27T15:53:31.8691517Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:31.8713792Z 
2022-09-27T15:53:31.8714242Z Running tests...
2022-09-27T15:53:31.8714714Z ----------------------------------------------------------------------
2022-09-27T15:53:31.9295471Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:31.9296075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:31.9298419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:31.9298895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:32.4651112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:32.4674671Z 
2022-09-27T15:53:32.4675093Z Running tests...
2022-09-27T15:53:32.4675567Z ----------------------------------------------------------------------
2022-09-27T15:53:32.4697187Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-09-27T15:53:32.4697569Z 
2022-09-27T15:53:32.4697905Z ----------------------------------------------------------------------
2022-09-27T15:53:32.4698268Z Ran 1 test in 0.002s
2022-09-27T15:53:32.4698438Z 
2022-09-27T15:53:32.4698551Z OK (skipped=1)
2022-09-27T15:53:32.4698691Z 
2022-09-27T15:53:32.4698820Z Generating XML reports...
2022-09-27T15:53:32.4734115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155332.xml
2022-09-27T15:53:33.1500913Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:33.1704745Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13778
2022-09-27T15:53:33.1710664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13779
2022-09-27T15:53:33.4453855Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:33.4737428Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13812
2022-09-27T15:53:33.4745384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13813
2022-09-27T15:53:34.6765598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:34.6766207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:34.6767454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:34.6767949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:35.0552096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:35.0552605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:35.0554831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:35.0555339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:35.0592986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:35.0593462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:35.0597114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:35.0597612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:35.1673795Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:35.1696409Z 
2022-09-27T15:53:35.1696772Z Running tests...
2022-09-27T15:53:35.1697226Z ----------------------------------------------------------------------
2022-09-27T15:53:35.1716602Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:35.1716938Z 
2022-09-27T15:53:35.1717206Z ----------------------------------------------------------------------
2022-09-27T15:53:35.1717542Z Ran 1 test in 0.002s
2022-09-27T15:53:35.1717715Z 
2022-09-27T15:53:35.1717827Z OK (skipped=1)
2022-09-27T15:53:35.1717987Z 
2022-09-27T15:53:35.1718114Z Generating XML reports...
2022-09-27T15:53:35.1753356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155335.xml
2022-09-27T15:53:35.3523860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:35.3524387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:35.3527184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:35.3527686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:35.3529258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:35.3529720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:35.3533926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:35.3534397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:35.5829359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:35.5836510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:35.5838227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:35.5931178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:35.5937686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:35.6852749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:35.8931035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:35.8932090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:35.8933324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:35.8934055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:35.9039687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:35.9107390Z STAGE:2022-09-27 15:53:35 13813:13813 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:35.9945094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:36.0010808Z STAGE:2022-09-27 15:53:36 13812:13812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:36.0028156Z STAGE:2022-09-27 15:53:36 13812:13812 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:36.0029283Z STAGE:2022-09-27 15:53:36 13813:13813 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:36.0056480Z STAGE:2022-09-27 15:53:36 13812:13812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:36.0057160Z STAGE:2022-09-27 15:53:36 13813:13813 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:36.0071932Z STAGE:2022-09-27 15:53:36 13813:13813 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:36.0073177Z STAGE:2022-09-27 15:53:36 13812:13812 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:36.0792030Z skip: Skipped due to small world size. (4.544s)
2022-09-27T15:53:36.0792570Z 
2022-09-27T15:53:36.0793379Z ----------------------------------------------------------------------
2022-09-27T15:53:36.0794134Z Ran 1 test in 4.544s
2022-09-27T15:53:36.0794503Z 
2022-09-27T15:53:36.0794741Z OK (skipped=1)
2022-09-27T15:53:36.0795093Z 
2022-09-27T15:53:36.0795344Z Generating XML reports...
2022-09-27T15:53:36.0859851Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155331.xml
2022-09-27T15:53:36.3815497Z ok (4.510s)
2022-09-27T15:53:36.3815709Z 
2022-09-27T15:53:36.3816379Z ----------------------------------------------------------------------
2022-09-27T15:53:36.3816745Z Ran 1 test in 4.510s
2022-09-27T15:53:36.3816930Z 
2022-09-27T15:53:36.3817008Z OK
2022-09-27T15:53:36.3817143Z 
2022-09-27T15:53:36.3817280Z Generating XML reports...
2022-09-27T15:53:36.3852840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155331.xml
2022-09-27T15:53:37.1078768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:37.1079737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:37.1081483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:37.1082451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:37.6694329Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:37.6718070Z 
2022-09-27T15:53:37.6718254Z Running tests...
2022-09-27T15:53:37.6719263Z ----------------------------------------------------------------------
2022-09-27T15:53:38.2527861Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:38.2528783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:38.2531838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:38.2532329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:38.4477386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:38.4477872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:38.4480523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:38.4481222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:38.7512512Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:53:38.7535282Z 
2022-09-27T15:53:38.7535540Z Running tests...
2022-09-27T15:53:38.7535977Z ----------------------------------------------------------------------
2022-09-27T15:53:38.9386144Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:38.9408520Z 
2022-09-27T15:53:38.9408746Z Running tests...
2022-09-27T15:53:38.9409200Z ----------------------------------------------------------------------
2022-09-27T15:53:39.2909696Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:39.3117876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14060
2022-09-27T15:53:39.3123854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14061
2022-09-27T15:53:40.3559933Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:40.3758419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14125
2022-09-27T15:53:40.3765123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14126
2022-09-27T15:53:40.5217713Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:40.5414895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14128
2022-09-27T15:53:40.5420034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14129
2022-09-27T15:53:40.9054213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:40.9054722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:40.9056766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:40.9057258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:40.9578912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:40.9579408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:40.9581662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:40.9582133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:41.5733906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:41.6290001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:41.6290872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:41.6342095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:41.6349316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:41.7303470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:42.0797798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:42.0798737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:42.0799642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:42.0800115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:42.1576390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:42.1576914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:42.1578680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:42.1579189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:42.2572024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:42.2572540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:42.2573589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:42.2574082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:42.2619665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:42.2620130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:42.2623380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:42.2623881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:42.6170093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:42.6935456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:42.6936319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:42.6981659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:42.6988909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:42.7058536Z STAGE:2022-09-27 15:53:42 14126:14126 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.7947753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:42.8012475Z STAGE:2022-09-27 15:53:42 14125:14125 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.8128257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:42.8128795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:42.8129551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:42.8130232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:42.8137613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:42.8138216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:42.8640802Z STAGE:2022-09-27 15:53:42 14128:14128 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.8642068Z STAGE:2022-09-27 15:53:42 14129:14129 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.8658730Z STAGE:2022-09-27 15:53:42 14128:14128 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.8659955Z STAGE:2022-09-27 15:53:42 14129:14129 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.8684780Z STAGE:2022-09-27 15:53:42 14128:14128 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.8693248Z STAGE:2022-09-27 15:53:42 14129:14129 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:42.8708575Z STAGE:2022-09-27 15:53:42 14128:14128 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:42.8709377Z STAGE:2022-09-27 15:53:42 14129:14129 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:43.4492481Z ok (4.508s)
2022-09-27T15:53:43.4492714Z 
2022-09-27T15:53:43.4493116Z ----------------------------------------------------------------------
2022-09-27T15:53:43.4493455Z Ran 1 test in 4.508s
2022-09-27T15:53:43.4493608Z 
2022-09-27T15:53:43.4493713Z OK
2022-09-27T15:53:43.4493853Z 
2022-09-27T15:53:43.4493989Z Generating XML reports...
2022-09-27T15:53:43.4529521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155338.xml
2022-09-27T15:53:44.4843721Z [1664294024.483801] [28fd6574525b:14125:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:53:44.4844639Z [1664294024.483801] [28fd6574525b:14125:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:53:44.5349930Z [1664294024.534497] [28fd6574525b:14126:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:53:44.5350463Z [1664294024.534497] [28fd6574525b:14126:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:53:44.8248507Z ok (7.152s)
2022-09-27T15:53:44.8248992Z 
2022-09-27T15:53:44.8249859Z ----------------------------------------------------------------------
2022-09-27T15:53:44.8250645Z Ran 1 test in 7.153s
2022-09-27T15:53:44.8251056Z 
2022-09-27T15:53:44.8251286Z OK
2022-09-27T15:53:44.8251612Z 
2022-09-27T15:53:44.8251933Z Generating XML reports...
2022-09-27T15:53:44.8284843Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155337.xml
2022-09-27T15:53:45.0914941Z STAGE:2022-09-27 15:53:45 14126:14126 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:45 14125:14125 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:45.0915431Z 
2022-09-27T15:53:45.2196859Z STAGE:2022-09-27 15:53:45 14125:14125 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:45.2201402Z STAGE:2022-09-27 15:53:45 14126:14126 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:45.2212444Z STAGE:2022-09-27 15:53:45 14125:14125 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:45 14126:14126 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:45.2212851Z 
2022-09-27T15:53:45.6623806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:45.6624306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:45.6627368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:45.6627857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:46.2281099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:46.2304570Z 
2022-09-27T15:53:46.2304765Z Running tests...
2022-09-27T15:53:46.2305211Z ----------------------------------------------------------------------
2022-09-27T15:53:46.3013156Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... ok (7.547s)
2022-09-27T15:53:46.3013460Z 
2022-09-27T15:53:46.3014099Z ----------------------------------------------------------------------
2022-09-27T15:53:46.3014413Z Ran 1 test in 7.547s
2022-09-27T15:53:46.3014579Z 
2022-09-27T15:53:46.3014680Z OK
2022-09-27T15:53:46.3014815Z 
2022-09-27T15:53:46.3014939Z Generating XML reports...
2022-09-27T15:53:46.3045880Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155338.xml
2022-09-27T15:53:46.9166373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:46.9166876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:46.9168739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:46.9169235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:47.4130064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:47.4153476Z 
2022-09-27T15:53:47.4153805Z Running tests...
2022-09-27T15:53:47.4154810Z ----------------------------------------------------------------------
2022-09-27T15:53:47.4177339Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:47.4178068Z 
2022-09-27T15:53:47.4178706Z ----------------------------------------------------------------------
2022-09-27T15:53:47.4179483Z Ran 1 test in 0.002s
2022-09-27T15:53:47.4179853Z 
2022-09-27T15:53:47.4180100Z OK (skipped=1)
2022-09-27T15:53:47.4180452Z 
2022-09-27T15:53:47.4180707Z Generating XML reports...
2022-09-27T15:53:47.4215595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155347.xml
2022-09-27T15:53:47.8504690Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:47.8706897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14404
2022-09-27T15:53:47.8714124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14405
2022-09-27T15:53:48.3492186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:48.3493552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:48.3500044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:48.3501279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:48.9483566Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:53:48.9505751Z 
2022-09-27T15:53:48.9505901Z Running tests...
2022-09-27T15:53:48.9506599Z ----------------------------------------------------------------------
2022-09-27T15:53:49.4417868Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:49.4418460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:49.4420945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:49.4421424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:49.5948392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:49.5948908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:49.5950366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:49.5951164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:49.6105394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:49.6105871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:49.6108744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:49.6109229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:49.9484848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:49.9508433Z 
2022-09-27T15:53:49.9508931Z Running tests...
2022-09-27T15:53:49.9509401Z ----------------------------------------------------------------------
2022-09-27T15:53:49.9528537Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:49.9528876Z 
2022-09-27T15:53:49.9529160Z ----------------------------------------------------------------------
2022-09-27T15:53:49.9529488Z Ran 1 test in 0.002s
2022-09-27T15:53:49.9529635Z 
2022-09-27T15:53:49.9529744Z OK (skipped=1)
2022-09-27T15:53:49.9529901Z 
2022-09-27T15:53:49.9530027Z Generating XML reports...
2022-09-27T15:53:49.9565668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155349.xml
2022-09-27T15:53:50.1529635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:50.1530489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:50.1531281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:50.1531996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:50.1538513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:50.1539017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:50.1638842Z STAGE:2022-09-27 15:53:50 14404:14404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:50.1639455Z STAGE:2022-09-27 15:53:50 14405:14405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:50.1658679Z STAGE:2022-09-27 15:53:50 14404:14404 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:50 14405:14405 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:50.1659123Z 
2022-09-27T15:53:50.1688956Z STAGE:2022-09-27 15:53:50 14404:14404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:50.1689563Z STAGE:2022-09-27 15:53:50 14405:14405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:50.1704309Z STAGE:2022-09-27 15:53:50 14405:14405 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:50 14404:14404 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:50.1704706Z 
2022-09-27T15:53:50.5147350Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:50.5346187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14518
2022-09-27T15:53:50.5351979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14519
2022-09-27T15:53:50.5777335Z ok (4.347s)
2022-09-27T15:53:50.5777541Z 
2022-09-27T15:53:50.5777953Z ----------------------------------------------------------------------
2022-09-27T15:53:50.5778284Z Ran 1 test in 4.347s
2022-09-27T15:53:50.5778455Z 
2022-09-27T15:53:50.5778551Z OK
2022-09-27T15:53:50.5778689Z 
2022-09-27T15:53:50.5778826Z Generating XML reports...
2022-09-27T15:53:50.5813669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155346.xml
2022-09-27T15:53:52.0559326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:52.0560001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:52.0561763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:52.0562347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:52.3091126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:52.3091973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:52.3093146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:52.3093788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:52.3899728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:52.3900451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:52.3901731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:52.3902544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:52.5547047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:52.5570506Z 
2022-09-27T15:53:52.5570837Z Running tests...
2022-09-27T15:53:52.5571789Z ----------------------------------------------------------------------
2022-09-27T15:53:52.5591140Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:53:52.5591801Z 
2022-09-27T15:53:52.5592413Z ----------------------------------------------------------------------
2022-09-27T15:53:52.5592904Z Ran 1 test in 0.002s
2022-09-27T15:53:52.5593070Z 
2022-09-27T15:53:52.5593163Z OK (skipped=1)
2022-09-27T15:53:52.5593318Z 
2022-09-27T15:53:52.5593442Z Generating XML reports...
2022-09-27T15:53:52.5628151Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155352.xml
2022-09-27T15:53:52.8119559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:52.8747219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:52.8748333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:52.8750858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:52.8751699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:52.9007332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:52.9008466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:52.9030794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:52.9037691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:52.9103609Z STAGE:2022-09-27 15:53:52 14518:14518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.0019996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:53.0085259Z STAGE:2022-09-27 15:53:53 14519:14519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:53.3758538Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:53:53.3782076Z 
2022-09-27T15:53:53.3782780Z Running tests...
2022-09-27T15:53:53.3783679Z ----------------------------------------------------------------------
2022-09-27T15:53:54.5231082Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:54.5231859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:54.5234095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:54.5234664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:54.7041965Z [1664294034.703687] [28fd6574525b:14518:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:53:54.7042514Z [1664294034.703687] [28fd6574525b:14518:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:53:54.7393210Z [1664294034.738876] [28fd6574525b:14519:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:53:54.7393724Z [1664294034.738876] [28fd6574525b:14519:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:53:55.0538898Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:55.0737976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14699
2022-09-27T15:53:55.0743394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14700
2022-09-27T15:53:55.0748985Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:53:55.0775033Z 
2022-09-27T15:53:55.0775352Z Running tests...
2022-09-27T15:53:55.0775780Z ----------------------------------------------------------------------
2022-09-27T15:53:55.1197247Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 15:53:55 14519:14519 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:53:55 14518:14518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:55.1197782Z 
2022-09-27T15:53:55.2469360Z STAGE:2022-09-27 15:53:55 14518:14518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:55.2473761Z STAGE:2022-09-27 15:53:55 14519:14519 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:53:55.2484806Z STAGE:2022-09-27 15:53:55 14518:14518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:55.2485710Z STAGE:2022-09-27 15:53:55 14519:14519 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:53:56.4479071Z ok (7.497s)
2022-09-27T15:53:56.4479310Z 
2022-09-27T15:53:56.4479716Z ----------------------------------------------------------------------
2022-09-27T15:53:56.4480074Z Ran 1 test in 7.497s
2022-09-27T15:53:56.4480559Z 
2022-09-27T15:53:56.4480637Z OK
2022-09-27T15:53:56.4480782Z 
2022-09-27T15:53:56.4480918Z Generating XML reports...
2022-09-27T15:53:56.4515200Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155348.xml
2022-09-27T15:53:56.7142702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:56.7143212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:56.7144600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:56.7145104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:56.7374914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:56.7375744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:56.7377108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:56.7377584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:56.8254699Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:53:56.8463485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14767
2022-09-27T15:53:56.8469355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14768
2022-09-27T15:53:57.2653112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:57.2654235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:57.2655032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:57.2655753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:57.2660089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:57.2660783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:57.7815114Z ok (4.403s)
2022-09-27T15:53:57.7815408Z 
2022-09-27T15:53:57.7816132Z ----------------------------------------------------------------------
2022-09-27T15:53:57.7816801Z Ran 1 test in 4.403s
2022-09-27T15:53:57.7816957Z 
2022-09-27T15:53:57.7817053Z OK
2022-09-27T15:53:57.7817187Z 
2022-09-27T15:53:57.7817329Z Generating XML reports...
2022-09-27T15:53:57.7871430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155353.xml
2022-09-27T15:53:58.5151334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:58.5151901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:58.5153340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:58.5154076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:58.5381951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:58.5382644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:58.5385351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:58.5386420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:58.6205374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:58.6215526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:58.6216858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:58.6217471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:53:59.0775153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:53:59.1051516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:53:59.1052332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:59.1078451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:53:59.1085013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:53:59.1646525Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:53:59.1669069Z 
2022-09-27T15:53:59.1669488Z Running tests...
2022-09-27T15:53:59.1669996Z ----------------------------------------------------------------------
2022-09-27T15:53:59.2065599Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:53:59.9285351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:53:59.9285885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:53:59.9287248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:53:59.9287723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:00.4671323Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:00.4696512Z 
2022-09-27T15:54:00.4696661Z Running tests...
2022-09-27T15:54:00.4697296Z ----------------------------------------------------------------------
2022-09-27T15:54:00.7370512Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 15:54:00 14768:14768 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:00.8239225Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:00.8390680Z STAGE:2022-09-27 15:54:00 14767:14767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:00.8439511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14922
2022-09-27T15:54:00.8446316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14923
2022-09-27T15:54:00.9293853Z STAGE:2022-09-27 15:54:00 14767:14767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:00.9294671Z STAGE:2022-09-27 15:54:00 14768:14768 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.0990596Z STAGE:2022-09-27 15:54:01 14768:14768 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:01.1020747Z STAGE:2022-09-27 15:54:01 14767:14767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:01.1036403Z STAGE:2022-09-27 15:54:01 14768:14768 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.1037876Z STAGE:2022-09-27 15:54:01 14767:14767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.1058525Z STAGE:2022-09-27 15:54:01 14768:14768 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:01.1060217Z STAGE:2022-09-27 15:54:01 14767:14767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:01.1071899Z STAGE:2022-09-27 15:54:01 14768:14768 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.1072555Z STAGE:2022-09-27 15:54:01 14767:14767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:01.7577127Z ok (6.680s)
2022-09-27T15:54:01.7577362Z 
2022-09-27T15:54:01.7577757Z ----------------------------------------------------------------------
2022-09-27T15:54:01.7578098Z Ran 1 test in 6.680s
2022-09-27T15:54:01.7578274Z 
2022-09-27T15:54:01.7578374Z OK
2022-09-27T15:54:01.7578514Z 
2022-09-27T15:54:01.7578655Z Generating XML reports...
2022-09-27T15:54:01.7612550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155355.xml
2022-09-27T15:54:02.1890380Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:02.2096940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14994
2022-09-27T15:54:02.2103112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14995
2022-09-27T15:54:02.4820609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:02.4822027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:02.4823233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:02.4824166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:02.5460043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:02.5461127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:02.5462426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:02.5463470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:03.0484140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:03.1327073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:03.1327921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:03.1396369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:03.1404323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:03.2341182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:03.8388891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:03.8389868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:03.8391004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:03.8391913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:03.8769183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:03.8770180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:03.8771355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:03.8772288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:03.9034784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:03.9036268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:03.9037508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:03.9038468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:04.4185895Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:54:04.4208459Z 
2022-09-27T15:54:04.4208920Z Running tests...
2022-09-27T15:54:04.4209427Z ----------------------------------------------------------------------
2022-09-27T15:54:04.4444024Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:04.4444690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:04.4445463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:04.4446177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:04.4554327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:04.5461966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:04.5572587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:54:04.5573575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:54:04.5574978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:04.5576390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:04.5642100Z STAGE:2022-09-27 15:54:04 14994:14994 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:04.5736082Z STAGE:2022-09-27 15:54:04 14995:14995 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:04.5749736Z STAGE:2022-09-27 15:54:04 14995:14995 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:04.5750901Z STAGE:2022-09-27 15:54:04 14994:14994 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:04.5767768Z STAGE:2022-09-27 15:54:04 14995:14995 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:04.5768935Z STAGE:2022-09-27 15:54:04 14994:14994 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:04.5783358Z STAGE:2022-09-27 15:54:04 14994:14994 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:04 14995:14995 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:04.5784153Z 
2022-09-27T15:54:04.7172267Z STAGE:2022-09-27 15:54:04 14923:14923 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:04.8355141Z STAGE:2022-09-27 15:54:04 14922:14922 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:04.8506087Z [1664294044.850063] [28fd6574525b:14922:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:04.8507074Z [1664294044.850063] [28fd6574525b:14922:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:04.8525860Z [1664294044.852221] [28fd6574525b:14923:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:04.8526966Z [1664294044.852221] [28fd6574525b:14923:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:05.0172753Z ok (4.547s)
2022-09-27T15:54:05.0173257Z 
2022-09-27T15:54:05.0174584Z ----------------------------------------------------------------------
2022-09-27T15:54:05.0175414Z Ran 1 test in 4.547s
2022-09-27T15:54:05.0175826Z 
2022-09-27T15:54:05.0176056Z OK
2022-09-27T15:54:05.0176395Z 
2022-09-27T15:54:05.0176741Z Generating XML reports...
2022-09-27T15:54:05.0211540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155400.xml
2022-09-27T15:54:05.3212495Z STAGE:2022-09-27 15:54:05 14923:14923 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:05.3213755Z STAGE:2022-09-27 15:54:05 14922:14922 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:05.4484621Z STAGE:2022-09-27 15:54:05 14923:14923 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:05.4500341Z STAGE:2022-09-27 15:54:05 14923:14923 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:05.4543214Z STAGE:2022-09-27 15:54:05 14922:14922 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:05.4558766Z STAGE:2022-09-27 15:54:05 14922:14922 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:06.0419848Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:06.0621704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15133
2022-09-27T15:54:06.0627354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15134
2022-09-27T15:54:06.1560409Z ok (6.989s)
2022-09-27T15:54:06.1561130Z 
2022-09-27T15:54:06.1561672Z ----------------------------------------------------------------------
2022-09-27T15:54:06.1562056Z Ran 1 test in 6.989s
2022-09-27T15:54:06.1562229Z 
2022-09-27T15:54:06.1562329Z OK
2022-09-27T15:54:06.1562470Z 
2022-09-27T15:54:06.1562604Z Generating XML reports...
2022-09-27T15:54:06.1605691Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155359.xml
2022-09-27T15:54:07.2618079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:07.2618637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:07.2621041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:07.2621544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:07.7753174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:07.7753688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:07.7755001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:07.7755475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:07.8004267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:07.8004743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:07.8008516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:07.8009006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:07.8247890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:07.8272344Z 
2022-09-27T15:54:07.8272599Z Running tests...
2022-09-27T15:54:07.8273033Z ----------------------------------------------------------------------
2022-09-27T15:54:08.2836603Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:08.3303015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:08.3304067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:08.3343466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:08.3350324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:08.3419734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:08.3420198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:08.3424337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:08.3424830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:08.4316866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:08.8910771Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:54:08.8935189Z 
2022-09-27T15:54:08.8935543Z Running tests...
2022-09-27T15:54:08.8936409Z ----------------------------------------------------------------------
2022-09-27T15:54:09.5018810Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:09.5612593Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15275
2022-09-27T15:54:09.5617307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15276
2022-09-27T15:54:09.9759109Z STAGE:2022-09-27 15:54:09 15133:15133 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:10.0526230Z STAGE:2022-09-27 15:54:10 15134:15134 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:10.1225691Z STAGE:2022-09-27 15:54:10 15134:15134 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:10.1226348Z STAGE:2022-09-27 15:54:10 15133:15133 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:10.3570797Z STAGE:2022-09-27 15:54:10 15133:15133 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:10.3613688Z STAGE:2022-09-27 15:54:10 15134:15134 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:10.3629434Z STAGE:2022-09-27 15:54:10 15133:15133 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:10.3630956Z STAGE:2022-09-27 15:54:10 15134:15134 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:10.3651809Z STAGE:2022-09-27 15:54:10 15133:15133 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:10.3653529Z STAGE:2022-09-27 15:54:10 15134:15134 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:10.3664105Z STAGE:2022-09-27 15:54:10 15133:15133 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:10.3666387Z STAGE:2022-09-27 15:54:10 15134:15134 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:10.5666074Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:10.5874168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15351
2022-09-27T15:54:10.5879250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15352
2022-09-27T15:54:10.9734662Z ok (6.552s)
2022-09-27T15:54:10.9735023Z 
2022-09-27T15:54:10.9735462Z ----------------------------------------------------------------------
2022-09-27T15:54:10.9735790Z Ran 1 test in 6.553s
2022-09-27T15:54:10.9736016Z 
2022-09-27T15:54:10.9736189Z OK
2022-09-27T15:54:10.9736473Z 
2022-09-27T15:54:10.9736614Z Generating XML reports...
2022-09-27T15:54:10.9771429Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155404.xml
2022-09-27T15:54:11.2696116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:11.2698369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:11.2700669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:11.2701793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:11.3497729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:11.3499360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:11.3503902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:11.3505827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:11.8948179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:11.8948738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:11.8949537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:11.8950485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:11.8956311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:11.8956816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:11.9167781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:54:11.9168326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:54:11.9169259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:11.9169962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:11.9249776Z STAGE:2022-09-27 15:54:11 15276:15276 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:11.9255323Z STAGE:2022-09-27 15:54:11 15275:15275 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:11.9270652Z STAGE:2022-09-27 15:54:11 15276:15276 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:11 15275:15275 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:11.9271071Z 
2022-09-27T15:54:11.9287765Z STAGE:2022-09-27 15:54:11 15276:15276 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:54:11 15275:15275 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:11.9288169Z 
2022-09-27T15:54:11.9301609Z STAGE:2022-09-27 15:54:11 15276:15276 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:11 15275:15275 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:11.9302018Z 
2022-09-27T15:54:12.2650396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:12.2650933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:12.2652086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:12.2652616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:12.3518782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:12.3519966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:12.3521059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:12.3521577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:12.3686511Z ok (4.541s)
2022-09-27T15:54:12.3686696Z 
2022-09-27T15:54:12.3687245Z ----------------------------------------------------------------------
2022-09-27T15:54:12.3687700Z Ran 1 test in 4.541s
2022-09-27T15:54:12.3687878Z 
2022-09-27T15:54:12.3687955Z OK
2022-09-27T15:54:12.3688096Z 
2022-09-27T15:54:12.3688228Z Generating XML reports...
2022-09-27T15:54:12.3725170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155407.xml
2022-09-27T15:54:12.7726194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:12.8716418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:12.8717249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:12.8739263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:12.8746252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:12.9729496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:13.1132371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:13.1132911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:13.1133726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:13.1134214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:13.6260638Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:54:13.6283611Z 
2022-09-27T15:54:13.6284377Z Running tests...
2022-09-27T15:54:13.6285279Z ----------------------------------------------------------------------
2022-09-27T15:54:14.5055300Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:14.5056457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:14.5057611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:14.5058500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:14.5173779Z STAGE:2022-09-27 15:54:14 15352:15352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:14.5880077Z STAGE:2022-09-27 15:54:14 15351:15351 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:14.6027497Z [1664294054.602227] [28fd6574525b:15351:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:14.6028530Z [1664294054.602227] [28fd6574525b:15351:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:14.6123954Z [1664294054.611959] [28fd6574525b:15352:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:14.6124498Z [1664294054.611959] [28fd6574525b:15352:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:15.0259017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:15.0282041Z 
2022-09-27T15:54:15.0282631Z Running tests...
2022-09-27T15:54:15.0283635Z ----------------------------------------------------------------------
2022-09-27T15:54:15.0294169Z STAGE:2022-09-27 15:54:15 15351:15351 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:15.0295412Z STAGE:2022-09-27 15:54:15 15352:15352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:15.1582111Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 15:54:15 15352:15352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:15.1595363Z STAGE:2022-09-27 15:54:15 15351:15351 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:15.1597173Z STAGE:2022-09-27 15:54:15 15352:15352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:15.1610946Z STAGE:2022-09-27 15:54:15 15351:15351 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:15.2172229Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:15.2369062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15523
2022-09-27T15:54:15.2374669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15524
2022-09-27T15:54:15.9015354Z ok (7.007s)
2022-09-27T15:54:15.9017336Z 
2022-09-27T15:54:15.9018075Z ----------------------------------------------------------------------
2022-09-27T15:54:15.9019221Z Ran 1 test in 7.008s
2022-09-27T15:54:15.9019557Z 
2022-09-27T15:54:15.9019737Z OK
2022-09-27T15:54:15.9020014Z 
2022-09-27T15:54:15.9020273Z Generating XML reports...
2022-09-27T15:54:15.9074819Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155408.xml
2022-09-27T15:54:16.7461155Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:16.7659088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15589
2022-09-27T15:54:16.7664264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15590
2022-09-27T15:54:16.8738787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:16.8739333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:16.8741209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:16.8741718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:16.9235015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:16.9235533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:16.9237756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:16.9238244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:17.3767014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:17.5467402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:17.5468300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:17.5490989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:17.5499097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:17.6480348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:18.1828531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:18.1829530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:18.1830738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:18.1832097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:18.4218707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:18.4219739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:18.4220947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:18.4221890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:18.4874279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:18.4875260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:18.4876744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:18.4877709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:18.7075728Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:54:18.7099042Z 
2022-09-27T15:54:18.7099728Z Running tests...
2022-09-27T15:54:18.7100616Z ----------------------------------------------------------------------
2022-09-27T15:54:19.0372669Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:19.0373928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:19.0375376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:19.0376779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:19.0484648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:19.1388168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:19.1449819Z STAGE:2022-09-27 15:54:19 15524:15524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.1499335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:54:19.1499878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:54:19.1500645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:19.1501330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:19.1599241Z STAGE:2022-09-27 15:54:19 15590:15590 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.1599856Z STAGE:2022-09-27 15:54:19 15589:15589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.1614418Z STAGE:2022-09-27 15:54:19 15589:15589 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:19 15590:15590 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:19.1614842Z 
2022-09-27T15:54:19.1631392Z STAGE:2022-09-27 15:54:19 15590:15590 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.1631987Z STAGE:2022-09-27 15:54:19 15589:15589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.1646698Z STAGE:2022-09-27 15:54:19 15589:15589 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:19 15590:15590 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:19.1647109Z 
2022-09-27T15:54:19.2328986Z STAGE:2022-09-27 15:54:19 15523:15523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.3992453Z STAGE:2022-09-27 15:54:19 15523:15523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:19.3993658Z STAGE:2022-09-27 15:54:19 15524:15524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:19.5644823Z STAGE:2022-09-27 15:54:19 15523:15523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.5672637Z STAGE:2022-09-27 15:54:19 15524:15524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.5689528Z STAGE:2022-09-27 15:54:19 15523:15523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:19.5690922Z STAGE:2022-09-27 15:54:19 15524:15524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:19.5712707Z STAGE:2022-09-27 15:54:19 15523:15523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.5714188Z STAGE:2022-09-27 15:54:19 15524:15524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:19.5727923Z STAGE:2022-09-27 15:54:19 15523:15523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:19.5729255Z STAGE:2022-09-27 15:54:19 15524:15524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:19.5730266Z ok (4.545s)
2022-09-27T15:54:19.5730576Z 
2022-09-27T15:54:19.5731081Z ----------------------------------------------------------------------
2022-09-27T15:54:19.5731715Z Ran 1 test in 4.545s
2022-09-27T15:54:19.5732022Z 
2022-09-27T15:54:19.5732197Z OK
2022-09-27T15:54:19.5732450Z 
2022-09-27T15:54:19.5732662Z Generating XML reports...
2022-09-27T15:54:19.5768767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155415.xml
2022-09-27T15:54:20.1483680Z ok (6.520s)
2022-09-27T15:54:20.1484211Z 
2022-09-27T15:54:20.1484632Z ----------------------------------------------------------------------
2022-09-27T15:54:20.1485112Z Ran 1 test in 6.520s
2022-09-27T15:54:20.1485438Z 
2022-09-27T15:54:20.1485617Z OK
2022-09-27T15:54:20.1485863Z 
2022-09-27T15:54:20.1486094Z Generating XML reports...
2022-09-27T15:54:20.1522585Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155413.xml
2022-09-27T15:54:20.4137791Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:20.4345392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15729
2022-09-27T15:54:20.4350433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15730
2022-09-27T15:54:21.9151620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:21.9152146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:21.9154131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:21.9154650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:22.1894130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:22.1894631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:22.1897179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:22.1897678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:22.1941469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:22.1941922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:22.1945802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:22.1946314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:22.3267716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:22.3268279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:22.3269177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:22.3269662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:22.4085426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:22.4109600Z 
2022-09-27T15:54:22.4109742Z Running tests...
2022-09-27T15:54:22.4110571Z ----------------------------------------------------------------------
2022-09-27T15:54:22.6971452Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:22.7044642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:22.7046188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:22.7072835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:22.7079958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:22.7148128Z STAGE:2022-09-27 15:54:22 15730:15730 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:22.8058394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:22.8123151Z STAGE:2022-09-27 15:54:22 15729:15729 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:22.8249061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:54:22.8272310Z 
2022-09-27T15:54:22.8272708Z Running tests...
2022-09-27T15:54:22.8273226Z ----------------------------------------------------------------------
2022-09-27T15:54:22.8292118Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:54:22.8292454Z 
2022-09-27T15:54:22.8292748Z ----------------------------------------------------------------------
2022-09-27T15:54:22.8293070Z Ran 1 test in 0.002s
2022-09-27T15:54:22.8293239Z 
2022-09-27T15:54:22.8293351Z OK (skipped=1)
2022-09-27T15:54:22.8293515Z 
2022-09-27T15:54:22.8293645Z Generating XML reports...
2022-09-27T15:54:22.8327921Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155422.xml
2022-09-27T15:54:24.1493843Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:24.1700663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15904
2022-09-27T15:54:24.1706946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15905
2022-09-27T15:54:24.4614603Z [1664294064.460946] [28fd6574525b:15730:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:24.4615139Z [1664294064.460946] [28fd6574525b:15730:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:24.4959835Z [1664294064.495557] [28fd6574525b:15729:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:24.4960638Z [1664294064.495557] [28fd6574525b:15729:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:24.7997705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:24.8000956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:24.8004405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:24.8007096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:24.9306896Z STAGE:2022-09-27 15:54:24 15730:15730 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:24.9307563Z STAGE:2022-09-27 15:54:24 15729:15729 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:25.0573272Z STAGE:2022-09-27 15:54:25 15730:15730 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:25.0963666Z STAGE:2022-09-27 15:54:25 15729:15729 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:25.0973925Z STAGE:2022-09-27 15:54:25 15729:15729 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:25 15730:15730 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:25.0974331Z 
2022-09-27T15:54:25.3504491Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:54:25.3526786Z 
2022-09-27T15:54:25.3527252Z Running tests...
2022-09-27T15:54:25.3527720Z ----------------------------------------------------------------------
2022-09-27T15:54:25.3545063Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:54:25.3545373Z 
2022-09-27T15:54:25.3545664Z ----------------------------------------------------------------------
2022-09-27T15:54:25.3545982Z Ran 1 test in 0.002s
2022-09-27T15:54:25.3546154Z 
2022-09-27T15:54:25.3546263Z OK (skipped=1)
2022-09-27T15:54:25.3546420Z 
2022-09-27T15:54:25.3546546Z Generating XML reports...
2022-09-27T15:54:25.3580138Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155425.xml
2022-09-27T15:54:25.8395901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:25.8396905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:25.8398086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:25.8399067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:25.8616532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:25.8617480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:25.8619854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:25.8620826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:26.2493943Z ok (7.539s)
2022-09-27T15:54:26.2494405Z 
2022-09-27T15:54:26.2495286Z ----------------------------------------------------------------------
2022-09-27T15:54:26.2496037Z Ran 1 test in 7.539s
2022-09-27T15:54:26.2496442Z 
2022-09-27T15:54:26.2496645Z OK
2022-09-27T15:54:26.2496986Z 
2022-09-27T15:54:26.2497290Z Generating XML reports...
2022-09-27T15:54:26.2561440Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155418.xml
2022-09-27T15:54:26.3817416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:26.3817988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:26.3818791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:26.3819484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:26.3927819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:26.4831402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:26.4943974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:54:26.4945091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:54:26.4946499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:26.4947971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:26.5043427Z STAGE:2022-09-27 15:54:26 15905:15905 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:26.5058393Z STAGE:2022-09-27 15:54:26 15904:15904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:26.5125850Z STAGE:2022-09-27 15:54:26 15905:15905 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:26 15904:15904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:26.5126276Z 
2022-09-27T15:54:26.5143282Z STAGE:2022-09-27 15:54:26 15904:15904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:26.5151937Z STAGE:2022-09-27 15:54:26 15905:15905 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:26.5167179Z STAGE:2022-09-27 15:54:26 15904:15904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:26.5168411Z STAGE:2022-09-27 15:54:26 15905:15905 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:26.9773409Z ok (4.566s)
2022-09-27T15:54:26.9773631Z 
2022-09-27T15:54:26.9774025Z ----------------------------------------------------------------------
2022-09-27T15:54:26.9774351Z Ran 1 test in 4.566s
2022-09-27T15:54:26.9774522Z 
2022-09-27T15:54:26.9774615Z OK
2022-09-27T15:54:26.9774750Z 
2022-09-27T15:54:26.9774888Z Generating XML reports...
2022-09-27T15:54:26.9810144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155422.xml
2022-09-27T15:54:27.2975631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:27.2976187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:27.2978459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:27.2978948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:27.8790558Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:54:27.8813984Z 
2022-09-27T15:54:27.8814280Z Running tests...
2022-09-27T15:54:27.8814698Z ----------------------------------------------------------------------
2022-09-27T15:54:28.4847009Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:28.4847611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:28.4849041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:28.4849534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:28.9756957Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:54:28.9779296Z 
2022-09-27T15:54:28.9779448Z Running tests...
2022-09-27T15:54:28.9780291Z ----------------------------------------------------------------------
2022-09-27T15:54:28.9808179Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T15:54:28.9808816Z 
2022-09-27T15:54:28.9809554Z ----------------------------------------------------------------------
2022-09-27T15:54:28.9809923Z Ran 1 test in 0.003s
2022-09-27T15:54:28.9810098Z 
2022-09-27T15:54:28.9810193Z OK (skipped=1)
2022-09-27T15:54:28.9810351Z 
2022-09-27T15:54:28.9810486Z Generating XML reports...
2022-09-27T15:54:28.9843131Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155428.xml
2022-09-27T15:54:29.1257545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:29.1258027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:29.1259585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:29.1260078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:29.5177808Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:29.5377626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16096
2022-09-27T15:54:29.5383333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16097
2022-09-27T15:54:29.6202214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:29.6224477Z 
2022-09-27T15:54:29.6224939Z Running tests...
2022-09-27T15:54:29.6225410Z ----------------------------------------------------------------------
2022-09-27T15:54:31.0333452Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:31.0334060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:31.0335395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:31.0335893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:31.2521404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:31.2521921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:31.2524299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:31.2524781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:31.2945211Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:31.3137387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16195
2022-09-27T15:54:31.3143535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16196
2022-09-27T15:54:31.3211144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:31.3211639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:31.3214891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:31.3215389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:31.5394983Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:54:31.5418905Z 
2022-09-27T15:54:31.5419231Z Running tests...
2022-09-27T15:54:31.5419667Z ----------------------------------------------------------------------
2022-09-27T15:54:31.7678166Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:31.8396581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:31.8397392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:31.8489236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:31.8496071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:31.9587858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:32.9547787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:32.9548347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:32.9549236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:32.9549731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:32.9786247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:32.9786769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:32.9789024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:32.9789843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:33.2574841Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:33.2784918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16273
2022-09-27T15:54:33.2790241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16274
2022-09-27T15:54:33.5329141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:33.5329684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:33.5330478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:33.5331199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:33.5336974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:33.5337681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:33.9227906Z skip: Skipped due to small world size. (4.299s)
2022-09-27T15:54:33.9229230Z 
2022-09-27T15:54:33.9230697Z ----------------------------------------------------------------------
2022-09-27T15:54:33.9231501Z Ran 1 test in 4.300s
2022-09-27T15:54:33.9231890Z 
2022-09-27T15:54:33.9232149Z OK (skipped=1)
2022-09-27T15:54:33.9232520Z 
2022-09-27T15:54:33.9232780Z Generating XML reports...
2022-09-27T15:54:33.9296980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155429.xml
2022-09-27T15:54:34.9515725Z ok (7.070s)
2022-09-27T15:54:34.9515978Z 
2022-09-27T15:54:34.9516380Z ----------------------------------------------------------------------
2022-09-27T15:54:34.9516732Z Ran 1 test in 7.070s
2022-09-27T15:54:34.9516899Z 
2022-09-27T15:54:34.9516994Z OK
2022-09-27T15:54:34.9517144Z 
2022-09-27T15:54:34.9518591Z Generating XML reports...
2022-09-27T15:54:34.9553209Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155427.xml
2022-09-27T15:54:34.9833465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:34.9834003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:34.9835410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:34.9835901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:34.9884636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:34.9885212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:34.9888069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:34.9888589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:35.5510982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:35.5787428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:35.5788274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:35.5814397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:35.5821292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:35.5886475Z STAGE:2022-09-27 15:54:35 16274:16274 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:35.6798615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:35.6864355Z STAGE:2022-09-27 15:54:35 16273:16273 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:36.1651492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:36.1657174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:36.1659194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:36.1660729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:36.7106375Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:36.7130777Z 
2022-09-27T15:54:36.7131102Z Running tests...
2022-09-27T15:54:36.7131568Z ----------------------------------------------------------------------
2022-09-27T15:54:37.1448026Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:37.1448597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:37.1449584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:37.1450079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:37.3871267Z [1664294077.386624] [28fd6574525b:16274:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:37.3871806Z [1664294077.386624] [28fd6574525b:16274:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:37.4434377Z [1664294077.442985] [28fd6574525b:16273:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:37.4434898Z [1664294077.442985] [28fd6574525b:16273:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:37.6592755Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:54:37.6616171Z 
2022-09-27T15:54:37.6616496Z Running tests...
2022-09-27T15:54:37.6616963Z ----------------------------------------------------------------------
2022-09-27T15:54:37.8471029Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 15:54:37 16273:16273 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:37 16274:16274 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:37.8471660Z 
2022-09-27T15:54:37.9719389Z STAGE:2022-09-27 15:54:37 16274:16274 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:37.9742995Z STAGE:2022-09-27 15:54:37 16273:16273 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:37.9753335Z STAGE:2022-09-27 15:54:37 16274:16274 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:37.9753945Z STAGE:2022-09-27 15:54:37 16273:16273 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:38.3468948Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:38.3668594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16440
2022-09-27T15:54:38.3674698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16441
2022-09-27T15:54:39.1924581Z ok (7.650s)
2022-09-27T15:54:39.1924792Z 
2022-09-27T15:54:39.1925177Z ----------------------------------------------------------------------
2022-09-27T15:54:39.1925522Z Ran 1 test in 7.651s
2022-09-27T15:54:39.1925693Z 
2022-09-27T15:54:39.1925807Z OK
2022-09-27T15:54:39.1925945Z 
2022-09-27T15:54:39.1926068Z Generating XML reports...
2022-09-27T15:54:39.1977556Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155431.xml
2022-09-27T15:54:39.3397135Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:39.3600301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16505
2022-09-27T15:54:39.3605655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16506
2022-09-27T15:54:40.0460252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:40.0461292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:40.0469300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:40.0471327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:40.1207024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:40.1207636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:40.1212579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:40.1213462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:40.7393060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:40.7393659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:40.7394455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:40.7395189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:40.7500780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:40.8406993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:41.0886887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:41.0887600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:41.0889226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:41.0889724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:41.1720820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:41.1721343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:41.1724154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:41.1724678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:41.1739902Z skip: Skipped due to small world size. (4.461s)
2022-09-27T15:54:41.1740125Z 
2022-09-27T15:54:41.1740712Z ----------------------------------------------------------------------
2022-09-27T15:54:41.1741238Z Ran 1 test in 4.461s
2022-09-27T15:54:41.1741472Z 
2022-09-27T15:54:41.1741646Z OK (skipped=1)
2022-09-27T15:54:41.1741791Z 
2022-09-27T15:54:41.1741918Z Generating XML reports...
2022-09-27T15:54:41.1778532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155436.xml
2022-09-27T15:54:41.5149907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:41.5150455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:41.5153004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:41.5153829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:41.6007040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:41.6955248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:41.6956063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:41.7019225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:41.7026844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:41.7968422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:42.0195725Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:54:42.0219455Z 
2022-09-27T15:54:42.0219793Z Running tests...
2022-09-27T15:54:42.0220250Z ----------------------------------------------------------------------
2022-09-27T15:54:43.2982676Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:43.2983267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:43.2984758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:43.2985257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:43.6909835Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:43.7111400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16660
2022-09-27T15:54:43.7116635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16661
2022-09-27T15:54:43.8173254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:43.8195432Z 
2022-09-27T15:54:43.8195685Z Running tests...
2022-09-27T15:54:43.8196110Z ----------------------------------------------------------------------
2022-09-27T15:54:45.2734823Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... ok (7.611s)
2022-09-27T15:54:45.2735336Z 
2022-09-27T15:54:45.2735735Z ----------------------------------------------------------------------
2022-09-27T15:54:45.2736070Z Ran 1 test in 7.612s
2022-09-27T15:54:45.2736373Z 
2022-09-27T15:54:45.2736546Z OK
2022-09-27T15:54:45.2736699Z 
2022-09-27T15:54:45.2736829Z Generating XML reports...
2022-09-27T15:54:45.2770144Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155437.xml
2022-09-27T15:54:45.4039681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:45.4040558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:45.4041590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:45.4042339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:45.4370167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:45.4370667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:45.4371940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:45.4372432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:45.5772050Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:45.5972981Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16735
2022-09-27T15:54:45.5979316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16736
2022-09-27T15:54:45.9336366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:45.9510693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:45.9511536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:45.9538471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:45.9545389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:45.9611249Z STAGE:2022-09-27 15:54:45 16661:16661 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:46.0524001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:46.0589569Z STAGE:2022-09-27 15:54:46 16660:16660 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:47.3325871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:47.3326408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:47.3327904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:47.3328384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.3866845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:47.3867487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:47.3868458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:47.3868932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.4806787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:47.4807291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:47.4808779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:47.4809252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:47.8250164Z [1664294087.824442] [28fd6574525b:16661:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:47.8250954Z [1664294087.824442] [28fd6574525b:16661:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:47.8443086Z [1664294087.843840] [28fd6574525b:16660:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:47.8443622Z [1664294087.843840] [28fd6574525b:16660:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:47.9305062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:47.9305813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:47.9306635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:47.9307330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:47.9412828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:47.9988103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:54:48.0011974Z 
2022-09-27T15:54:48.0012649Z Running tests...
2022-09-27T15:54:48.0013079Z ----------------------------------------------------------------------
2022-09-27T15:54:48.0031046Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:54:48.0031626Z 
2022-09-27T15:54:48.0032128Z ----------------------------------------------------------------------
2022-09-27T15:54:48.0032589Z Ran 1 test in 0.002s
2022-09-27T15:54:48.0032758Z 
2022-09-27T15:54:48.0032852Z OK (skipped=1)
2022-09-27T15:54:48.0033010Z 
2022-09-27T15:54:48.0033138Z Generating XML reports...
2022-09-27T15:54:48.0067819Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155447.xml
2022-09-27T15:54:48.0320008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:48.2445248Z STAGE:2022-09-27 15:54:48 16661:16661 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:48 16660:16660 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:48.2445724Z 
2022-09-27T15:54:48.3692256Z STAGE:2022-09-27 15:54:48 16661:16661 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:48.3785852Z STAGE:2022-09-27 15:54:48 16660:16660 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:48.3796210Z STAGE:2022-09-27 15:54:48 16660:16660 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:48.3797383Z STAGE:2022-09-27 15:54:48 16661:16661 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:48.4049699Z skip: Skipped due to small world size. (4.585s)
2022-09-27T15:54:48.4050136Z 
2022-09-27T15:54:48.4050725Z ----------------------------------------------------------------------
2022-09-27T15:54:48.4051276Z Ran 1 test in 4.585s
2022-09-27T15:54:48.4051540Z 
2022-09-27T15:54:48.4051716Z OK (skipped=1)
2022-09-27T15:54:48.4051973Z 
2022-09-27T15:54:48.4052163Z Generating XML reports...
2022-09-27T15:54:48.4087573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155443.xml
2022-09-27T15:54:49.5244139Z ok (7.502s)
2022-09-27T15:54:49.5244531Z 
2022-09-27T15:54:49.5245199Z ----------------------------------------------------------------------
2022-09-27T15:54:49.5245832Z Ran 1 test in 7.502s
2022-09-27T15:54:49.5246151Z 
2022-09-27T15:54:49.5246325Z OK
2022-09-27T15:54:49.5246582Z 
2022-09-27T15:54:49.5246824Z Generating XML reports...
2022-09-27T15:54:49.5281050Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155442.xml
2022-09-27T15:54:50.1131705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:50.1132705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:50.1133862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:50.1134836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:50.5979624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:50.5980603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:50.5982737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:50.5983668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:50.6072750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:54:50.6096952Z 
2022-09-27T15:54:50.6097868Z Running tests...
2022-09-27T15:54:50.6098794Z ----------------------------------------------------------------------
2022-09-27T15:54:51.1571903Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:51.1596605Z 
2022-09-27T15:54:51.1597271Z Running tests...
2022-09-27T15:54:51.1598190Z ----------------------------------------------------------------------
2022-09-27T15:54:51.6307861Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:51.6308945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:51.6309998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:51.6310995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:52.1347204Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:54:52.1370444Z 
2022-09-27T15:54:52.1371111Z Running tests...
2022-09-27T15:54:52.1371651Z ----------------------------------------------------------------------
2022-09-27T15:54:52.2412962Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:52.2615706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16960
2022-09-27T15:54:52.2621054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16961
2022-09-27T15:54:52.7958837Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:52.8166191Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16963
2022-09-27T15:54:52.8170933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16964
2022-09-27T15:54:53.8382959Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:54:53.8580406Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17090
2022-09-27T15:54:53.8586145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17091
2022-09-27T15:54:53.9650483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:53.9650984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:53.9654376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:53.9654877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:53.9691606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:53.9692082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:53.9696501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:53.9696996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:54.4558391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:54.4558914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:54.4561011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:54.4561517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:54.4819552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:54.4822445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:54.4830494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:54.4845052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:54.5023911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:54.5035271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:54:54.5221412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:54.5222896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:54.5229416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:54.5230917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:54.5871244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:54.5873815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:54:54.5876775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:54.5961676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:54:55.0835252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:55.0835866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:55.0836675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:55.0837401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:55.0945068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:55.1850210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:55.5532861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:55.5533369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:55.5535616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:55.5536115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:55.5635249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:55.5635737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:55.5639036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:55.5639548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:55.6245139Z skip: Skipped due to small world size. (4.464s)
2022-09-27T15:54:55.6245386Z 
2022-09-27T15:54:55.6245748Z ----------------------------------------------------------------------
2022-09-27T15:54:55.6246067Z Ran 1 test in 4.465s
2022-09-27T15:54:55.6246241Z 
2022-09-27T15:54:55.6246356Z OK (skipped=1)
2022-09-27T15:54:55.6246516Z 
2022-09-27T15:54:55.6246653Z Generating XML reports...
2022-09-27T15:54:55.6283284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155451.xml
2022-09-27T15:54:56.0914212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:54:56.1234263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:54:56.1235114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:56.1320425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:54:56.1327554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:54:56.1438513Z STAGE:2022-09-27 15:54:56 17091:17091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:56.2249296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:54:56.2314599Z STAGE:2022-09-27 15:54:56 17090:17090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:57.7743735Z ok (7.164s)
2022-09-27T15:54:57.7743962Z 
2022-09-27T15:54:57.7744367Z ----------------------------------------------------------------------
2022-09-27T15:54:57.7744731Z Ran 1 test in 7.165s
2022-09-27T15:54:57.7744884Z 
2022-09-27T15:54:57.7744981Z OK
2022-09-27T15:54:57.7745124Z 
2022-09-27T15:54:57.7745259Z Generating XML reports...
2022-09-27T15:54:57.7780034Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155450.xml
2022-09-27T15:54:57.9168592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:57.9169079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:57.9170435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:57.9170924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:54:58.0139946Z [1664294098.013467] [28fd6574525b:17090:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:58.0140506Z [1664294098.013467] [28fd6574525b:17090:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:58.0339840Z [1664294098.033503] [28fd6574525b:17091:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:54:58.0340362Z [1664294098.033503] [28fd6574525b:17091:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:54:58.4245013Z STAGE:2022-09-27 15:54:58 17091:17091 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:54:58 17090:17090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:58.4245459Z 
2022-09-27T15:54:58.4329953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:54:58.4353353Z 
2022-09-27T15:54:58.4353562Z Running tests...
2022-09-27T15:54:58.4354004Z ----------------------------------------------------------------------
2022-09-27T15:54:58.5424788Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 15:54:58 17091:17091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:58.5475271Z STAGE:2022-09-27 15:54:58 17090:17090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:54:58.5485918Z STAGE:2022-09-27 15:54:58 17091:17091 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:58.5486537Z STAGE:2022-09-27 15:54:58 17090:17090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:54:59.6719670Z ok (7.535s)
2022-09-27T15:54:59.6719900Z 
2022-09-27T15:54:59.6720498Z ----------------------------------------------------------------------
2022-09-27T15:54:59.6720832Z Ran 1 test in 7.535s
2022-09-27T15:54:59.6721001Z 
2022-09-27T15:54:59.6721095Z OK
2022-09-27T15:54:59.6721229Z 
2022-09-27T15:54:59.6721375Z Generating XML reports...
2022-09-27T15:54:59.6773063Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155452.xml
2022-09-27T15:54:59.9261557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:54:59.9262437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:54:59.9264938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:54:59.9265429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:00.1239820Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:00.1437613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17274
2022-09-27T15:55:00.1444016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17275
2022-09-27T15:55:00.4289877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:00.4312095Z 
2022-09-27T15:55:00.4312355Z Running tests...
2022-09-27T15:55:00.4312830Z ----------------------------------------------------------------------
2022-09-27T15:55:00.4331693Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:55:00.4332238Z 
2022-09-27T15:55:00.4332686Z ----------------------------------------------------------------------
2022-09-27T15:55:00.4333046Z Ran 1 test in 0.002s
2022-09-27T15:55:00.4333220Z 
2022-09-27T15:55:00.4333335Z OK (skipped=1)
2022-09-27T15:55:00.4333476Z 
2022-09-27T15:55:00.4333606Z Generating XML reports...
2022-09-27T15:55:00.4367944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155500.xml
2022-09-27T15:55:01.7974449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:01.7975368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:01.7976482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:01.7977404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:01.8015733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:01.8016647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:01.8019956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:01.8020947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:01.8152630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:01.8153593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:01.8157186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:01.8158221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:02.3010721Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:02.3034588Z 
2022-09-27T15:55:02.3035066Z Running tests...
2022-09-27T15:55:02.3035543Z ----------------------------------------------------------------------
2022-09-27T15:55:02.3082875Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.005s)
2022-09-27T15:55:02.3083219Z 
2022-09-27T15:55:02.3083748Z ----------------------------------------------------------------------
2022-09-27T15:55:02.3084140Z Ran 1 test in 0.005s
2022-09-27T15:55:02.3084292Z 
2022-09-27T15:55:02.3084408Z OK (skipped=1)
2022-09-27T15:55:02.3084568Z 
2022-09-27T15:55:02.3084695Z Generating XML reports...
2022-09-27T15:55:02.3120428Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155502.xml
2022-09-27T15:55:02.3566286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:02.3567206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:02.3567992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:02.3568701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:02.3576619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:02.3577117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:02.3674573Z STAGE:2022-09-27 15:55:02 17274:17274 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:02.3675205Z STAGE:2022-09-27 15:55:02 17275:17275 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:02.3689976Z STAGE:2022-09-27 15:55:02 17275:17275 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:55:02 17274:17274 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:02.3690397Z 
2022-09-27T15:55:02.3707189Z STAGE:2022-09-27 15:55:02 17275:17275 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:55:02 17274:17274 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:02.3707600Z 
2022-09-27T15:55:02.3720909Z STAGE:2022-09-27 15:55:02 17274:17274 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:55:02 17275:17275 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:02.3721317Z 
2022-09-27T15:55:02.3828931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:02.3829421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:02.3832834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:02.3833343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:02.7511738Z ok (4.315s)
2022-09-27T15:55:02.7511977Z 
2022-09-27T15:55:02.7512380Z ----------------------------------------------------------------------
2022-09-27T15:55:02.7512721Z Ran 1 test in 4.316s
2022-09-27T15:55:02.7512892Z 
2022-09-27T15:55:02.7512968Z OK
2022-09-27T15:55:02.7513110Z 
2022-09-27T15:55:02.7513244Z Generating XML reports...
2022-09-27T15:55:02.7548101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155458.xml
2022-09-27T15:55:02.8967793Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:02.8990549Z 
2022-09-27T15:55:02.8991112Z Running tests...
2022-09-27T15:55:02.8991568Z ----------------------------------------------------------------------
2022-09-27T15:55:04.3391648Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:04.3392277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:04.3393850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:04.3394344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:04.5352821Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:04.5546359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17484
2022-09-27T15:55:04.5552270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17485
2022-09-27T15:55:04.8239808Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:04.8262344Z 
2022-09-27T15:55:04.8262844Z Running tests...
2022-09-27T15:55:04.8263366Z ----------------------------------------------------------------------
2022-09-27T15:55:04.8283903Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.002s)
2022-09-27T15:55:04.8284697Z 
2022-09-27T15:55:04.8285023Z ----------------------------------------------------------------------
2022-09-27T15:55:04.8285388Z Ran 1 test in 0.002s
2022-09-27T15:55:04.8285691Z 
2022-09-27T15:55:04.8285890Z OK (skipped=1)
2022-09-27T15:55:04.8286066Z 
2022-09-27T15:55:04.8286196Z Generating XML reports...
2022-09-27T15:55:04.8319829Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155504.xml
2022-09-27T15:55:04.9551119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:04.9551640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:04.9552971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:04.9553495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:05.5418509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:05.5440485Z 
2022-09-27T15:55:05.5440744Z Running tests...
2022-09-27T15:55:05.5441422Z ----------------------------------------------------------------------
2022-09-27T15:55:06.2670260Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:06.2670862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:06.2672966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:06.2673445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:06.2714286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:06.2714754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:06.2718409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:06.2718879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:06.7868775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:06.7908806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:06.7909847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:06.7969510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:06.7976501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:06.8336390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:06.8336889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:06.8338941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:06.8339414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:06.8924422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:07.1836734Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:07.2031089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17592
2022-09-27T15:55:07.2037332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17593
2022-09-27T15:55:07.2614090Z skip: Skipped due to small world size. (4.362s)
2022-09-27T15:55:07.2614366Z 
2022-09-27T15:55:07.2614753Z ----------------------------------------------------------------------
2022-09-27T15:55:07.2615097Z Ran 1 test in 4.362s
2022-09-27T15:55:07.2615264Z 
2022-09-27T15:55:07.2615381Z OK (skipped=1)
2022-09-27T15:55:07.2615538Z 
2022-09-27T15:55:07.2615648Z Generating XML reports...
2022-09-27T15:55:07.2651424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155502.xml
2022-09-27T15:55:07.3603217Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:07.3626300Z 
2022-09-27T15:55:07.3626718Z Running tests...
2022-09-27T15:55:07.3627489Z ----------------------------------------------------------------------
2022-09-27T15:55:07.3650707Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.002s)
2022-09-27T15:55:07.3651367Z 
2022-09-27T15:55:07.3651681Z ----------------------------------------------------------------------
2022-09-27T15:55:07.3652022Z Ran 1 test in 0.002s
2022-09-27T15:55:07.3652230Z 
2022-09-27T15:55:07.3652435Z OK (skipped=1)
2022-09-27T15:55:07.3652682Z 
2022-09-27T15:55:07.3652815Z Generating XML reports...
2022-09-27T15:55:07.3686633Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155507.xml
2022-09-27T15:55:08.9431866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:08.9432374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:08.9433749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:08.9434355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:08.9698988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:08.9699537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:08.9702787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:08.9703541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:09.3804232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:09.3804762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:09.3807632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:09.3808160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:09.4316144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:09.4316620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:09.4320493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:09.4321209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:09.4928213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:09.4928754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:09.4929530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:09.4930232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:09.4937009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:09.4937513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:09.5034843Z STAGE:2022-09-27 15:55:09 17593:17593 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:09.5035455Z STAGE:2022-09-27 15:55:09 17592:17592 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:09.5049536Z STAGE:2022-09-27 15:55:09 17592:17592 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:55:09 17593:17593 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:09.5049948Z 
2022-09-27T15:55:09.5066309Z STAGE:2022-09-27 15:55:09 17592:17592 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:55:09 17593:17593 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:09.5066703Z 
2022-09-27T15:55:09.5081540Z STAGE:2022-09-27 15:55:09 17593:17593 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:55:09 17592:17592 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:09.5081940Z 
2022-09-27T15:55:09.8885971Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:09.8908998Z 
2022-09-27T15:55:09.8909188Z Running tests...
2022-09-27T15:55:09.8909631Z ----------------------------------------------------------------------
2022-09-27T15:55:09.8928261Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:55:09.8928755Z 
2022-09-27T15:55:09.8929220Z ----------------------------------------------------------------------
2022-09-27T15:55:09.8929608Z Ran 1 test in 0.002s
2022-09-27T15:55:09.8929783Z 
2022-09-27T15:55:09.8929910Z OK (skipped=1)
2022-09-27T15:55:09.8930073Z 
2022-09-27T15:55:09.8930206Z Generating XML reports...
2022-09-27T15:55:09.8965384Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155509.xml
2022-09-27T15:55:09.9102310Z ok (4.366s)
2022-09-27T15:55:09.9102493Z 
2022-09-27T15:55:09.9102807Z ----------------------------------------------------------------------
2022-09-27T15:55:09.9103151Z Ran 1 test in 4.366s
2022-09-27T15:55:09.9103319Z 
2022-09-27T15:55:09.9103417Z OK
2022-09-27T15:55:09.9103535Z 
2022-09-27T15:55:09.9103666Z Generating XML reports...
2022-09-27T15:55:09.9140604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155505.xml
2022-09-27T15:55:09.9288989Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:09.9312700Z 
2022-09-27T15:55:09.9313233Z Running tests...
2022-09-27T15:55:09.9313671Z ----------------------------------------------------------------------
2022-09-27T15:55:09.9332176Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:09.9333196Z 
2022-09-27T15:55:09.9333485Z ----------------------------------------------------------------------
2022-09-27T15:55:09.9333826Z Ran 1 test in 0.002s
2022-09-27T15:55:09.9333997Z 
2022-09-27T15:55:09.9334096Z OK (skipped=1)
2022-09-27T15:55:09.9334450Z 
2022-09-27T15:55:09.9334714Z Generating XML reports...
2022-09-27T15:55:09.9369709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155509.xml
2022-09-27T15:55:11.9121605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:11.9122134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:11.9122754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:11.9123546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:11.9267882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:11.9268337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:11.9271889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:11.9272377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:12.0842745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:12.0843227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:12.0846418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:12.0846910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:12.4011534Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:12.4033354Z 
2022-09-27T15:55:12.4033484Z Running tests...
2022-09-27T15:55:12.4034481Z ----------------------------------------------------------------------
2022-09-27T15:55:12.4051659Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:12.4052188Z 
2022-09-27T15:55:12.4052832Z ----------------------------------------------------------------------
2022-09-27T15:55:12.4053527Z Ran 1 test in 0.002s
2022-09-27T15:55:12.4053870Z 
2022-09-27T15:55:12.4054088Z OK (skipped=1)
2022-09-27T15:55:12.4054368Z 
2022-09-27T15:55:12.4054610Z Generating XML reports...
2022-09-27T15:55:12.4087990Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155512.xml
2022-09-27T15:55:12.4230714Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:12.4254885Z 
2022-09-27T15:55:12.4255336Z Running tests...
2022-09-27T15:55:12.4256216Z ----------------------------------------------------------------------
2022-09-27T15:55:12.4273744Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:55:12.4275051Z 
2022-09-27T15:55:12.4275625Z ----------------------------------------------------------------------
2022-09-27T15:55:12.4276307Z Ran 1 test in 0.002s
2022-09-27T15:55:12.4276611Z 
2022-09-27T15:55:12.4276827Z OK (skipped=1)
2022-09-27T15:55:12.4277140Z 
2022-09-27T15:55:12.4277381Z Generating XML reports...
2022-09-27T15:55:12.4312949Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155512.xml
2022-09-27T15:55:12.5780474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:12.5803981Z 
2022-09-27T15:55:12.5804457Z Running tests...
2022-09-27T15:55:12.5804963Z ----------------------------------------------------------------------
2022-09-27T15:55:14.1974594Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:14.2165827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17901
2022-09-27T15:55:14.2171984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17902
2022-09-27T15:55:14.3919264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:14.3919772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:14.3922231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:14.3922726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:14.4189905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:14.4190378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:14.4193358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:14.4193847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:14.9290252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:14.9332148Z 
2022-09-27T15:55:14.9332513Z Running tests...
2022-09-27T15:55:14.9334247Z ----------------------------------------------------------------------
2022-09-27T15:55:14.9621963Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:14.9645345Z 
2022-09-27T15:55:14.9645644Z Running tests...
2022-09-27T15:55:14.9646116Z ----------------------------------------------------------------------
2022-09-27T15:55:14.9664406Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:55:14.9665086Z 
2022-09-27T15:55:14.9665500Z ----------------------------------------------------------------------
2022-09-27T15:55:14.9666106Z Ran 1 test in 0.002s
2022-09-27T15:55:14.9666279Z 
2022-09-27T15:55:14.9666392Z OK (skipped=1)
2022-09-27T15:55:14.9666550Z 
2022-09-27T15:55:14.9666661Z Generating XML reports...
2022-09-27T15:55:14.9700561Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155514.xml
2022-09-27T15:55:15.8706645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:15.8707172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:15.8708214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:15.8708723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:15.8939058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:15.8940075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:15.8941234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:15.8942186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:16.4226111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:16.4227487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:16.4228870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:16.4229640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:16.4334922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:16.5242293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:16.5374016Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:16.5544627Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2022-09-27T15:55:16.5545673Z 
2022-09-27T15:55:16.5546189Z ----------------------------------------------------------------------
2022-09-27T15:55:16.5547138Z Ran 1 test in 1.621s
2022-09-27T15:55:16.5547430Z 
2022-09-27T15:55:16.5547618Z OK (skipped=1)
2022-09-27T15:55:16.5547892Z 
2022-09-27T15:55:16.5548091Z Generating XML reports...
2022-09-27T15:55:16.5579160Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155514.xml
2022-09-27T15:55:16.8620078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:16.8621097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:16.8622325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:16.8623334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:17.3738692Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:17.3761592Z 
2022-09-27T15:55:17.3762060Z Running tests...
2022-09-27T15:55:17.3762595Z ----------------------------------------------------------------------
2022-09-27T15:55:17.3782398Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:55:17.3782725Z 
2022-09-27T15:55:17.3783021Z ----------------------------------------------------------------------
2022-09-27T15:55:17.3783353Z Ran 1 test in 0.002s
2022-09-27T15:55:17.3783524Z 
2022-09-27T15:55:17.3783632Z OK (skipped=1)
2022-09-27T15:55:17.3783774Z 
2022-09-27T15:55:17.3783900Z Generating XML reports...
2022-09-27T15:55:17.3819162Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155517.xml
2022-09-27T15:55:18.1002723Z STAGE:2022-09-27 15:55:18 17902:17902 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:18.2109751Z STAGE:2022-09-27 15:55:18 17901:17901 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:18.2160145Z STAGE:2022-09-27 15:55:18 17901:17901 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:18.2161408Z STAGE:2022-09-27 15:55:18 17902:17902 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:18.2220835Z STAGE:2022-09-27 15:55:18 17901:17901 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:18.2362841Z STAGE:2022-09-27 15:55:18 17902:17902 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:18.2385386Z STAGE:2022-09-27 15:55:18 17901:17901 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:18.2385987Z STAGE:2022-09-27 15:55:18 17902:17902 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:18.7741052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:18.7741579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:18.7743831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:18.7744371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:18.8281222Z ok (6.248s)
2022-09-27T15:55:18.8281637Z 
2022-09-27T15:55:18.8282132Z ----------------------------------------------------------------------
2022-09-27T15:55:18.8282494Z Ran 1 test in 6.248s
2022-09-27T15:55:18.8282665Z 
2022-09-27T15:55:18.8282930Z OK
2022-09-27T15:55:18.8283064Z 
2022-09-27T15:55:18.8283196Z Generating XML reports...
2022-09-27T15:55:18.8338654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155512.xml
2022-09-27T15:55:19.2684032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:19.2706282Z 
2022-09-27T15:55:19.2706635Z Running tests...
2022-09-27T15:55:19.2707061Z ----------------------------------------------------------------------
2022-09-27T15:55:19.4343127Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:19.4343746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:19.4345103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:19.4345587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:19.9256166Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:19.9278622Z 
2022-09-27T15:55:19.9279205Z Running tests...
2022-09-27T15:55:19.9280707Z ----------------------------------------------------------------------
2022-09-27T15:55:19.9298855Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:55:19.9299773Z 
2022-09-27T15:55:19.9300664Z ----------------------------------------------------------------------
2022-09-27T15:55:19.9301474Z Ran 1 test in 0.002s
2022-09-27T15:55:19.9301875Z 
2022-09-27T15:55:19.9302131Z OK (skipped=1)
2022-09-27T15:55:19.9302469Z 
2022-09-27T15:55:19.9302759Z Generating XML reports...
2022-09-27T15:55:19.9338069Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155519.xml
2022-09-27T15:55:20.8384779Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:20.8817540Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18149
2022-09-27T15:55:20.8907804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18150
2022-09-27T15:55:20.9005254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:20.9006313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:20.9013905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:20.9015049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:21.4304313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:21.4327787Z 
2022-09-27T15:55:21.4328107Z Running tests...
2022-09-27T15:55:21.4328553Z ----------------------------------------------------------------------
2022-09-27T15:55:21.9135770Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:21.9136647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:21.9137590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:21.9138094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:22.4157268Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:22.4179739Z 
2022-09-27T15:55:22.4180033Z Running tests...
2022-09-27T15:55:22.4180858Z ----------------------------------------------------------------------
2022-09-27T15:55:22.4198594Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:55:22.4199436Z 
2022-09-27T15:55:22.4199737Z ----------------------------------------------------------------------
2022-09-27T15:55:22.4200069Z Ran 1 test in 0.002s
2022-09-27T15:55:22.4200531Z 
2022-09-27T15:55:22.4200652Z OK (skipped=1)
2022-09-27T15:55:22.4200814Z 
2022-09-27T15:55:22.4200964Z Generating XML reports...
2022-09-27T15:55:22.4235112Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155522.xml
2022-09-27T15:55:22.5738694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:22.5739581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:22.5740595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:22.5741400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:22.6332213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:22.6333072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:22.6334068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:22.6334858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:23.0396715Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:23.0589683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18220
2022-09-27T15:55:23.0596246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18221
2022-09-27T15:55:23.0973502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:23.1568284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:23.1569114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:23.1581680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:23.1588567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:23.2582441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:24.4649638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:24.4650140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:24.4652441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:24.4652938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:24.7721572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:24.7722078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:24.7724396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:24.7724924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:24.7759140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:24.7759606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:24.7763335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:24.7763826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:24.8398557Z STAGE:2022-09-27 15:55:24 18150:18150 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:24.9260072Z STAGE:2022-09-27 15:55:24 18149:18149 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:24.9881251Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:24.9904829Z 
2022-09-27T15:55:24.9904982Z Running tests...
2022-09-27T15:55:24.9905758Z ----------------------------------------------------------------------
2022-09-27T15:55:24.9924662Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:55:24.9924988Z 
2022-09-27T15:55:24.9925283Z ----------------------------------------------------------------------
2022-09-27T15:55:24.9925617Z Ran 1 test in 0.002s
2022-09-27T15:55:24.9925785Z 
2022-09-27T15:55:24.9925884Z OK (skipped=1)
2022-09-27T15:55:24.9926043Z 
2022-09-27T15:55:24.9926168Z Generating XML reports...
2022-09-27T15:55:24.9960714Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155524.xml
2022-09-27T15:55:25.3317840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:25.3318413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:25.3319200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:25.3319917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:25.3328142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:25.3328656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:26.3806979Z STAGE:2022-09-27 15:55:26 18149:18149 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:26.3809427Z STAGE:2022-09-27 15:55:26 18150:18150 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:27.0092567Z STAGE:2022-09-27 15:55:27 18221:18221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:27.0261400Z STAGE:2022-09-27 15:55:27 18220:18220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:27.0279142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:27.0279618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:27.0283381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:27.0283881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:27.0311250Z STAGE:2022-09-27 15:55:27 18220:18220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:27.0311834Z STAGE:2022-09-27 15:55:27 18221:18221 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:27.0372946Z STAGE:2022-09-27 15:55:27 18220:18220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:27.0377323Z STAGE:2022-09-27 15:55:27 18221:18221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:27.0400509Z STAGE:2022-09-27 15:55:27 18220:18220 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:55:27 18221:18221 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:27.0401135Z 
2022-09-27T15:55:27.5324888Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:27.5346764Z 
2022-09-27T15:55:27.5347193Z Running tests...
2022-09-27T15:55:27.5347704Z ----------------------------------------------------------------------
2022-09-27T15:55:27.5380606Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.003s)
2022-09-27T15:55:27.5380940Z 
2022-09-27T15:55:27.5381205Z ----------------------------------------------------------------------
2022-09-27T15:55:27.5381538Z Ran 1 test in 0.003s
2022-09-27T15:55:27.5381702Z 
2022-09-27T15:55:27.5381827Z OK (skipped=1)
2022-09-27T15:55:27.5381985Z 
2022-09-27T15:55:27.5382114Z Generating XML reports...
2022-09-27T15:55:27.5415959Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155527.xml
2022-09-27T15:55:27.5697581Z ok (6.137s)
2022-09-27T15:55:27.5697781Z 
2022-09-27T15:55:27.5698109Z ----------------------------------------------------------------------
2022-09-27T15:55:27.5698430Z Ran 1 test in 6.137s
2022-09-27T15:55:27.5698602Z 
2022-09-27T15:55:27.5698695Z OK
2022-09-27T15:55:27.5698834Z 
2022-09-27T15:55:27.5698967Z Generating XML reports...
2022-09-27T15:55:27.5734922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155521.xml
2022-09-27T15:55:28.2056218Z ok (8.935s)
2022-09-27T15:55:28.2056439Z 
2022-09-27T15:55:28.2056837Z ----------------------------------------------------------------------
2022-09-27T15:55:28.2057197Z Ran 1 test in 8.935s
2022-09-27T15:55:28.2057390Z 
2022-09-27T15:55:28.2057470Z OK
2022-09-27T15:55:28.2057608Z 
2022-09-27T15:55:28.2057741Z Generating XML reports...
2022-09-27T15:55:28.2093052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155519.xml
2022-09-27T15:55:29.5324669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:29.5325648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:29.5326830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:29.5327772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:29.6518585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:29.6519559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:29.6521115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:29.6522090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:30.0329042Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:30.0351715Z 
2022-09-27T15:55:30.0352193Z Running tests...
2022-09-27T15:55:30.0353043Z ----------------------------------------------------------------------
2022-09-27T15:55:30.0371178Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:30.0371862Z 
2022-09-27T15:55:30.0372426Z ----------------------------------------------------------------------
2022-09-27T15:55:30.0373086Z Ran 1 test in 0.002s
2022-09-27T15:55:30.0373351Z 
2022-09-27T15:55:30.0373562Z OK (skipped=1)
2022-09-27T15:55:30.0373849Z 
2022-09-27T15:55:30.0374463Z Generating XML reports...
2022-09-27T15:55:30.0407241Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155530.xml
2022-09-27T15:55:30.1448357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:30.1471099Z 
2022-09-27T15:55:30.1471797Z Running tests...
2022-09-27T15:55:30.1472659Z ----------------------------------------------------------------------
2022-09-27T15:55:30.2377678Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:30.2378817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:30.2380460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:30.2381419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:30.7368661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:30.7391977Z 
2022-09-27T15:55:30.7392451Z Running tests...
2022-09-27T15:55:30.7393284Z ----------------------------------------------------------------------
2022-09-27T15:55:30.7411898Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:30.7412448Z 
2022-09-27T15:55:30.7412899Z ----------------------------------------------------------------------
2022-09-27T15:55:30.7413250Z Ran 1 test in 0.002s
2022-09-27T15:55:30.7413401Z 
2022-09-27T15:55:30.7413519Z OK (skipped=1)
2022-09-27T15:55:30.7413680Z 
2022-09-27T15:55:30.7413814Z Generating XML reports...
2022-09-27T15:55:30.7448440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155530.xml
2022-09-27T15:55:31.7723913Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:31.8238747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18553
2022-09-27T15:55:31.8293346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18554
2022-09-27T15:55:32.0534586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:32.0535121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:32.0536028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:32.0536526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:32.6461107Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:32.6485322Z 
2022-09-27T15:55:32.6485723Z Running tests...
2022-09-27T15:55:32.6486263Z ----------------------------------------------------------------------
2022-09-27T15:55:32.6503705Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:32.6504264Z 
2022-09-27T15:55:32.6504629Z ----------------------------------------------------------------------
2022-09-27T15:55:32.6504977Z Ran 1 test in 0.002s
2022-09-27T15:55:32.6505154Z 
2022-09-27T15:55:32.6505269Z OK (skipped=1)
2022-09-27T15:55:32.6505412Z 
2022-09-27T15:55:32.6505538Z Generating XML reports...
2022-09-27T15:55:32.6540768Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155532.xml
2022-09-27T15:55:32.8308426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:32.8308920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:32.8311740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:32.8312262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:33.3238303Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:33.3260618Z 
2022-09-27T15:55:33.3261047Z Running tests...
2022-09-27T15:55:33.3261540Z ----------------------------------------------------------------------
2022-09-27T15:55:33.4658865Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:33.4659462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:33.4660492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:33.4660978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:33.5133876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:33.5134384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:33.5137359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:33.5137852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:34.0641160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:34.0641748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:34.0642556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:34.0643249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:34.0749955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:34.0818647Z STAGE:2022-09-27 15:55:34 18554:18554 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:34.1654380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:34.1723541Z STAGE:2022-09-27 15:55:34 18553:18553 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:34.1739519Z STAGE:2022-09-27 15:55:34 18553:18553 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:34.1740685Z STAGE:2022-09-27 15:55:34 18554:18554 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:34.1757336Z STAGE:2022-09-27 15:55:34 18553:18553 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:34.1758231Z STAGE:2022-09-27 15:55:34 18554:18554 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:34.1774124Z STAGE:2022-09-27 15:55:34 18553:18553 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:55:34 18554:18554 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:34.1774935Z 
2022-09-27T15:55:34.5367677Z ok (4.389s)
2022-09-27T15:55:34.5367910Z 
2022-09-27T15:55:34.5368311Z ----------------------------------------------------------------------
2022-09-27T15:55:34.5368657Z Ran 1 test in 4.390s
2022-09-27T15:55:34.5368809Z 
2022-09-27T15:55:34.5368906Z OK
2022-09-27T15:55:34.5369044Z 
2022-09-27T15:55:34.5369177Z Generating XML reports...
2022-09-27T15:55:34.5425762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155530.xml
2022-09-27T15:55:34.5650929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:34.5651466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:34.5653194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:34.5653706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:34.8780351Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:34.8942713Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.568s)
2022-09-27T15:55:34.8943452Z 
2022-09-27T15:55:34.8943873Z ----------------------------------------------------------------------
2022-09-27T15:55:34.8944200Z Ran 1 test in 1.568s
2022-09-27T15:55:34.8944373Z 
2022-09-27T15:55:34.8944485Z OK (skipped=1)
2022-09-27T15:55:34.8944717Z 
2022-09-27T15:55:34.8944949Z Generating XML reports...
2022-09-27T15:55:34.8978928Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155533.xml
2022-09-27T15:55:35.0753176Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:35.0776648Z 
2022-09-27T15:55:35.0777041Z Running tests...
2022-09-27T15:55:35.0777519Z ----------------------------------------------------------------------
2022-09-27T15:55:35.0796762Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:35.0797112Z 
2022-09-27T15:55:35.0797376Z ----------------------------------------------------------------------
2022-09-27T15:55:35.0797709Z Ran 1 test in 0.002s
2022-09-27T15:55:35.0797871Z 
2022-09-27T15:55:35.0797982Z OK (skipped=1)
2022-09-27T15:55:35.0798139Z 
2022-09-27T15:55:35.0798265Z Generating XML reports...
2022-09-27T15:55:35.0834167Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155535.xml
2022-09-27T15:55:36.7089778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:36.7090312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:36.7091525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:36.7092009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:37.0104306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:37.0104797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:37.0106719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:37.0107211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:37.0220747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:37.0221215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:37.0224884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:37.0225369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:37.2025571Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:37.2049435Z 
2022-09-27T15:55:37.2049883Z Running tests...
2022-09-27T15:55:37.2050399Z ----------------------------------------------------------------------
2022-09-27T15:55:37.5023527Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:37.5045968Z 
2022-09-27T15:55:37.5046717Z Running tests...
2022-09-27T15:55:37.5047203Z ----------------------------------------------------------------------
2022-09-27T15:55:37.5065976Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:37.5066350Z 
2022-09-27T15:55:37.5066637Z ----------------------------------------------------------------------
2022-09-27T15:55:37.5066970Z Ran 1 test in 0.002s
2022-09-27T15:55:37.5067133Z 
2022-09-27T15:55:37.5067245Z OK (skipped=1)
2022-09-27T15:55:37.5067383Z 
2022-09-27T15:55:37.5067513Z Generating XML reports...
2022-09-27T15:55:37.5102585Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155537.xml
2022-09-27T15:55:37.5252409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:37.5276592Z 
2022-09-27T15:55:37.5276895Z Running tests...
2022-09-27T15:55:37.5277361Z ----------------------------------------------------------------------
2022-09-27T15:55:37.5296821Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:37.5297554Z 
2022-09-27T15:55:37.5297838Z ----------------------------------------------------------------------
2022-09-27T15:55:37.5298172Z Ran 1 test in 0.002s
2022-09-27T15:55:37.5298336Z 
2022-09-27T15:55:37.5298443Z OK (skipped=1)
2022-09-27T15:55:37.5298598Z 
2022-09-27T15:55:37.5298706Z Generating XML reports...
2022-09-27T15:55:37.5334672Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155537.xml
2022-09-27T15:55:38.8487077Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:38.8685302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18832
2022-09-27T15:55:38.8691497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18833
2022-09-27T15:55:39.4788025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:39.4788561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:39.4791076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:39.4791587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:39.5260101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:39.5261015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:39.5265444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:39.5266397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:39.9661166Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:39.9684232Z 
2022-09-27T15:55:39.9684495Z Running tests...
2022-09-27T15:55:39.9684918Z ----------------------------------------------------------------------
2022-09-27T15:55:39.9702910Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:39.9703269Z 
2022-09-27T15:55:39.9703547Z ----------------------------------------------------------------------
2022-09-27T15:55:39.9703874Z Ran 1 test in 0.002s
2022-09-27T15:55:39.9704043Z 
2022-09-27T15:55:39.9704142Z OK (skipped=1)
2022-09-27T15:55:39.9704300Z 
2022-09-27T15:55:39.9704421Z Generating XML reports...
2022-09-27T15:55:39.9739949Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155539.xml
2022-09-27T15:55:40.0248043Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:40.0271911Z 
2022-09-27T15:55:40.0272057Z Running tests...
2022-09-27T15:55:40.0272521Z ----------------------------------------------------------------------
2022-09-27T15:55:40.4924436Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:40.4925069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:40.4927039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:40.4927536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:40.5116845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:40.5117353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:40.5119947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:40.5120704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:41.0526464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:41.0527546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:41.0529049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:41.0530461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:41.0634946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:41.1541283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:41.6184774Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:41.6385213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18944
2022-09-27T15:55:41.6391111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18945
2022-09-27T15:55:41.8832673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:41.8833647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:41.8834820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:41.8835730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:42.4582983Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:42.4605742Z 
2022-09-27T15:55:42.4606519Z Running tests...
2022-09-27T15:55:42.4607461Z ----------------------------------------------------------------------
2022-09-27T15:55:42.4625818Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:42.4626518Z 
2022-09-27T15:55:42.4627092Z ----------------------------------------------------------------------
2022-09-27T15:55:42.4627750Z Ran 1 test in 0.002s
2022-09-27T15:55:42.4628061Z 
2022-09-27T15:55:42.4628251Z OK (skipped=1)
2022-09-27T15:55:42.4628537Z 
2022-09-27T15:55:42.4628778Z Generating XML reports...
2022-09-27T15:55:42.4665362Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155542.xml
2022-09-27T15:55:42.9789908Z ok (5.774s)
2022-09-27T15:55:42.9790129Z 
2022-09-27T15:55:42.9790520Z ----------------------------------------------------------------------
2022-09-27T15:55:42.9790850Z Ran 1 test in 5.774s
2022-09-27T15:55:42.9791024Z 
2022-09-27T15:55:42.9791117Z OK
2022-09-27T15:55:42.9791251Z 
2022-09-27T15:55:42.9791634Z Generating XML reports...
2022-09-27T15:55:42.9827914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155537.xml
2022-09-27T15:55:43.2838022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:43.2838532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:43.2841139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:43.2841925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:43.2918377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:43.2918844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:43.2929467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:43.2929964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:43.7928520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:43.8571601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:43.8572457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:43.8638008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:43.8646489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:43.9585183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:44.3458339Z skip: Skipped due to small world size. (4.318s)
2022-09-27T15:55:44.3458828Z 
2022-09-27T15:55:44.3459491Z ----------------------------------------------------------------------
2022-09-27T15:55:44.3460117Z Ran 1 test in 4.318s
2022-09-27T15:55:44.3460438Z 
2022-09-27T15:55:44.3460641Z OK (skipped=1)
2022-09-27T15:55:44.3460937Z 
2022-09-27T15:55:44.3461172Z Generating XML reports...
2022-09-27T15:55:44.3497500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155540.xml
2022-09-27T15:55:44.4532731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:44.4533755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:44.4535044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:44.4536141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:44.9542495Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:44.9565224Z 
2022-09-27T15:55:44.9565648Z Running tests...
2022-09-27T15:55:44.9566168Z ----------------------------------------------------------------------
2022-09-27T15:55:44.9584052Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:44.9584407Z 
2022-09-27T15:55:44.9584696Z ----------------------------------------------------------------------
2022-09-27T15:55:44.9585019Z Ran 1 test in 0.002s
2022-09-27T15:55:44.9585187Z 
2022-09-27T15:55:44.9585299Z OK (skipped=1)
2022-09-27T15:55:44.9585462Z 
2022-09-27T15:55:44.9585592Z Generating XML reports...
2022-09-27T15:55:44.9620498Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155544.xml
2022-09-27T15:55:45.0529317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:45.0529848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:45.0531294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:45.0531774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:45.5471550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:45.5493799Z 
2022-09-27T15:55:45.5494075Z Running tests...
2022-09-27T15:55:45.5494781Z ----------------------------------------------------------------------
2022-09-27T15:55:46.4178582Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:46.4179505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:46.4180127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:46.4180619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:46.8271774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:46.8272308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:46.8274322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:46.8274833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:46.9250384Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:46.9274067Z 
2022-09-27T15:55:46.9274491Z Running tests...
2022-09-27T15:55:46.9275223Z ----------------------------------------------------------------------
2022-09-27T15:55:46.9293312Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:46.9294235Z 
2022-09-27T15:55:46.9294638Z ----------------------------------------------------------------------
2022-09-27T15:55:46.9295156Z Ran 1 test in 0.002s
2022-09-27T15:55:46.9295336Z 
2022-09-27T15:55:46.9295507Z OK (skipped=1)
2022-09-27T15:55:46.9295795Z 
2022-09-27T15:55:46.9295961Z Generating XML reports...
2022-09-27T15:55:46.9330068Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155546.xml
2022-09-27T15:55:47.1554672Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:47.1746760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19151
2022-09-27T15:55:47.1752872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19152
2022-09-27T15:55:47.3212049Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:47.3233872Z 
2022-09-27T15:55:47.3234084Z Running tests...
2022-09-27T15:55:47.3234850Z ----------------------------------------------------------------------
2022-09-27T15:55:47.3254273Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:47.3254969Z 
2022-09-27T15:55:47.3255501Z ----------------------------------------------------------------------
2022-09-27T15:55:47.3255923Z Ran 1 test in 0.002s
2022-09-27T15:55:47.3256073Z 
2022-09-27T15:55:47.3256185Z OK (skipped=1)
2022-09-27T15:55:47.3256343Z 
2022-09-27T15:55:47.3256473Z Generating XML reports...
2022-09-27T15:55:47.3290868Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155547.xml
2022-09-27T15:55:48.8368811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:48.8370223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:48.8371460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:48.8372399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:48.8886479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:48.8887447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:48.8888995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:48.8889932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:48.8952956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:48.8953927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:48.8956316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:48.8957158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:49.2815342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:49.2815826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:49.2818215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:49.2818712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:49.3841926Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:49.3864716Z 
2022-09-27T15:55:49.3864927Z Running tests...
2022-09-27T15:55:49.3865391Z ----------------------------------------------------------------------
2022-09-27T15:55:49.3883619Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:49.3883975Z 
2022-09-27T15:55:49.3884255Z ----------------------------------------------------------------------
2022-09-27T15:55:49.3884575Z Ran 1 test in 0.002s
2022-09-27T15:55:49.3884747Z 
2022-09-27T15:55:49.3884857Z OK (skipped=1)
2022-09-27T15:55:49.3885015Z 
2022-09-27T15:55:49.3885143Z Generating XML reports...
2022-09-27T15:55:49.3918937Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155549.xml
2022-09-27T15:55:49.4196760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:49.4197305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:49.4198053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:49.4198762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:49.4306132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:49.4373984Z STAGE:2022-09-27 15:55:49 19152:19152 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:49.5210901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:49.5276814Z STAGE:2022-09-27 15:55:49 19151:19151 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:49.5292381Z STAGE:2022-09-27 15:55:49 19151:19151 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:49.5292989Z STAGE:2022-09-27 15:55:49 19152:19152 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:49.5312128Z STAGE:2022-09-27 15:55:49 19151:19151 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:49.5314565Z STAGE:2022-09-27 15:55:49 19152:19152 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:49.5327496Z STAGE:2022-09-27 15:55:49 19151:19151 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:49.5328075Z STAGE:2022-09-27 15:55:49 19152:19152 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:49.7765159Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:49.7788575Z 
2022-09-27T15:55:49.7788822Z Running tests...
2022-09-27T15:55:49.7789261Z ----------------------------------------------------------------------
2022-09-27T15:55:49.7808360Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:49.7808702Z 
2022-09-27T15:55:49.7809002Z ----------------------------------------------------------------------
2022-09-27T15:55:49.7809340Z Ran 1 test in 0.002s
2022-09-27T15:55:49.7809508Z 
2022-09-27T15:55:49.7809619Z OK (skipped=1)
2022-09-27T15:55:49.7810076Z 
2022-09-27T15:55:49.7812007Z Generating XML reports...
2022-09-27T15:55:49.7844633Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155549.xml
2022-09-27T15:55:49.8817918Z ok (4.332s)
2022-09-27T15:55:49.8818236Z 
2022-09-27T15:55:49.8818897Z ----------------------------------------------------------------------
2022-09-27T15:55:49.8819524Z Ran 1 test in 4.332s
2022-09-27T15:55:49.8819815Z 
2022-09-27T15:55:49.8819973Z OK
2022-09-27T15:55:49.8820216Z 
2022-09-27T15:55:49.8820415Z Generating XML reports...
2022-09-27T15:55:49.8857589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155545.xml
2022-09-27T15:55:51.4048872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:51.4049391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:51.4051159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:51.4051648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:51.7808599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:51.7809106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:51.7810391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:51.7810860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:51.9035180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:55:51.9059042Z 
2022-09-27T15:55:51.9059315Z Running tests...
2022-09-27T15:55:51.9059756Z ----------------------------------------------------------------------
2022-09-27T15:55:52.0377114Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:52.0377742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:52.0379945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:52.0380445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:52.2650648Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:52.2672671Z 
2022-09-27T15:55:52.2672975Z Running tests...
2022-09-27T15:55:52.2673685Z ----------------------------------------------------------------------
2022-09-27T15:55:52.2692097Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:52.2692473Z 
2022-09-27T15:55:52.2692750Z ----------------------------------------------------------------------
2022-09-27T15:55:52.2693071Z Ran 1 test in 0.002s
2022-09-27T15:55:52.2693236Z 
2022-09-27T15:55:52.2693349Z OK (skipped=1)
2022-09-27T15:55:52.2693504Z 
2022-09-27T15:55:52.2693630Z Generating XML reports...
2022-09-27T15:55:52.2727189Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155552.xml
2022-09-27T15:55:52.5403366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:52.5426759Z 
2022-09-27T15:55:52.5427018Z Running tests...
2022-09-27T15:55:52.5427488Z ----------------------------------------------------------------------
2022-09-27T15:55:53.5656892Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:53.5854951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19429
2022-09-27T15:55:53.5859743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19430
2022-09-27T15:55:54.0808866Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:55:54.1009425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19432
2022-09-27T15:55:54.1015343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19433
2022-09-27T15:55:54.2288654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:54.2294628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:54.2299928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:54.2304834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:54.8091229Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:54.8138789Z 
2022-09-27T15:55:54.8140792Z Running tests...
2022-09-27T15:55:54.8143281Z ----------------------------------------------------------------------
2022-09-27T15:55:54.8190212Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.004s)
2022-09-27T15:55:54.8192662Z 
2022-09-27T15:55:54.8194752Z ----------------------------------------------------------------------
2022-09-27T15:55:54.8197012Z Ran 1 test in 0.004s
2022-09-27T15:55:54.8198820Z 
2022-09-27T15:55:54.8201144Z OK (skipped=1)
2022-09-27T15:55:54.8203014Z 
2022-09-27T15:55:54.8204777Z Generating XML reports...
2022-09-27T15:55:54.8260031Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155554.xml
2022-09-27T15:55:55.2838360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:55.2838942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:55.2840436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:55.2840950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:55.3138739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:55.3139244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:55.3141710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:55.3142278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:55.7874681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:55.7875263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:55.7880726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:55.7882306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:55.7951992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:55.8337438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:55.8339191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:55.8361363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:55.8371498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:55.8835177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:55.8836449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:55.8837577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:55.8838400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:55.9354088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:56.4268784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:55:56.4269748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:55:56.4270583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:56.4271300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:55:56.4276771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:55:56.4277855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:55:56.4380412Z STAGE:2022-09-27 15:55:56 19432:19432 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:55:56 19433:19433 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:56.4380900Z 
2022-09-27T15:55:56.4394608Z STAGE:2022-09-27 15:55:56 19433:19433 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:55:56 19432:19432 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:56.4395101Z 
2022-09-27T15:55:56.4412935Z STAGE:2022-09-27 15:55:56 19433:19433 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:55:56 19432:19432 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:55:56.4413462Z 
2022-09-27T15:55:56.4426909Z STAGE:2022-09-27 15:55:56 19433:19433 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:56.4427871Z STAGE:2022-09-27 15:55:56 19432:19432 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:55:56.7746916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:56.7747715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:56.7749070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:56.7749982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:56.9082785Z ok (4.365s)
2022-09-27T15:55:56.9083526Z 
2022-09-27T15:55:56.9084250Z ----------------------------------------------------------------------
2022-09-27T15:55:56.9084683Z Ran 1 test in 4.365s
2022-09-27T15:55:56.9084867Z 
2022-09-27T15:55:56.9084964Z OK
2022-09-27T15:55:56.9085101Z 
2022-09-27T15:55:56.9085238Z Generating XML reports...
2022-09-27T15:55:56.9118546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155552.xml
2022-09-27T15:55:57.3042295Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:57.3066552Z 
2022-09-27T15:55:57.3066931Z Running tests...
2022-09-27T15:55:57.3067419Z ----------------------------------------------------------------------
2022-09-27T15:55:57.3088079Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:55:57.3088441Z 
2022-09-27T15:55:57.3088843Z ----------------------------------------------------------------------
2022-09-27T15:55:57.3089286Z Ran 1 test in 0.002s
2022-09-27T15:55:57.3089471Z 
2022-09-27T15:55:57.3089676Z OK (skipped=1)
2022-09-27T15:55:57.3090198Z 
2022-09-27T15:55:57.3090330Z Generating XML reports...
2022-09-27T15:55:57.3126900Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155557.xml
2022-09-27T15:55:58.9977751Z ok (7.091s)
2022-09-27T15:55:58.9978102Z 
2022-09-27T15:55:58.9978750Z ----------------------------------------------------------------------
2022-09-27T15:55:58.9979112Z Ran 1 test in 7.092s
2022-09-27T15:55:58.9979266Z 
2022-09-27T15:55:58.9979359Z OK
2022-09-27T15:55:58.9979495Z 
2022-09-27T15:55:58.9979631Z Generating XML reports...
2022-09-27T15:55:59.0015314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155551.xml
2022-09-27T15:55:59.1017439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:59.1017945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:59.1020228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:59.1020728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:59.3570521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:55:59.3571046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:55:59.3573268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:55:59.3573768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:55:59.6082947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:55:59.6106853Z 
2022-09-27T15:55:59.6107180Z Running tests...
2022-09-27T15:55:59.6107622Z ----------------------------------------------------------------------
2022-09-27T15:55:59.8539029Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:55:59.8562305Z 
2022-09-27T15:55:59.8562567Z Running tests...
2022-09-27T15:55:59.8563014Z ----------------------------------------------------------------------
2022-09-27T15:55:59.8580904Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:55:59.8581472Z 
2022-09-27T15:55:59.8582142Z ----------------------------------------------------------------------
2022-09-27T15:55:59.8582819Z Ran 1 test in 0.002s
2022-09-27T15:55:59.8583135Z 
2022-09-27T15:55:59.8583362Z OK (skipped=1)
2022-09-27T15:55:59.8583674Z 
2022-09-27T15:55:59.8584298Z Generating XML reports...
2022-09-27T15:55:59.8618609Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155559.xml
2022-09-27T15:56:01.0915113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:01.0915843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:01.0916997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:01.0917509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:01.2914011Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:01.3113060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19759
2022-09-27T15:56:01.3119097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19760
2022-09-27T15:56:01.5922986Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:01.5945577Z 
2022-09-27T15:56:01.5945941Z Running tests...
2022-09-27T15:56:01.5946463Z ----------------------------------------------------------------------
2022-09-27T15:56:01.9688817Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:01.9691778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:01.9694777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:01.9697489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:02.5289920Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:02.5312250Z 
2022-09-27T15:56:02.5312675Z Running tests...
2022-09-27T15:56:02.5313213Z ----------------------------------------------------------------------
2022-09-27T15:56:02.5330887Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:56:02.5331275Z 
2022-09-27T15:56:02.5331548Z ----------------------------------------------------------------------
2022-09-27T15:56:02.5331895Z Ran 1 test in 0.002s
2022-09-27T15:56:02.5332065Z 
2022-09-27T15:56:02.5332184Z OK (skipped=1)
2022-09-27T15:56:02.5332344Z 
2022-09-27T15:56:02.5332473Z Generating XML reports...
2022-09-27T15:56:02.5366820Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155602.xml
2022-09-27T15:56:02.9327916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:02.9328450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:02.9330722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:02.9331207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:03.0149340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:03.0149858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:03.0151925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:03.0152404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:03.1977966Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:03.2172343Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19829
2022-09-27T15:56:03.2178622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19830
2022-09-27T15:56:03.5854882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:03.5860103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:03.5863903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:03.5957407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:03.6050041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:03.6155947Z STAGE:2022-09-27 15:56:03 19760:19760 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:03.6873591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:03.6941153Z STAGE:2022-09-27 15:56:03 19759:19759 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:03.6957095Z STAGE:2022-09-27 15:56:03 19759:19759 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:03.6957707Z STAGE:2022-09-27 15:56:03 19760:19760 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:03.6975251Z STAGE:2022-09-27 15:56:03 19759:19759 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:03.6975884Z STAGE:2022-09-27 15:56:03 19760:19760 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:03.6991041Z STAGE:2022-09-27 15:56:03 19760:19760 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:56:03 19759:19759 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:03.6991455Z 
2022-09-27T15:56:04.2191970Z ok (4.608s)
2022-09-27T15:56:04.2192175Z 
2022-09-27T15:56:04.2192556Z ----------------------------------------------------------------------
2022-09-27T15:56:04.2192918Z Ran 1 test in 4.608s
2022-09-27T15:56:04.2193106Z 
2022-09-27T15:56:04.2193205Z OK
2022-09-27T15:56:04.2193342Z 
2022-09-27T15:56:04.2193478Z Generating XML reports...
2022-09-27T15:56:04.2228117Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155559.xml
2022-09-27T15:56:04.6415968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:04.6416486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:04.6417856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:04.6418327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:04.9117081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:04.9117595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:04.9120269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:04.9120944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:04.9688403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:04.9688895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:04.9691276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:04.9691756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:05.1645463Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:05.1668187Z 
2022-09-27T15:56:05.1668463Z Running tests...
2022-09-27T15:56:05.1668917Z ----------------------------------------------------------------------
2022-09-27T15:56:05.1687942Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:56:05.1688324Z 
2022-09-27T15:56:05.1688615Z ----------------------------------------------------------------------
2022-09-27T15:56:05.1688960Z Ran 1 test in 0.002s
2022-09-27T15:56:05.1689127Z 
2022-09-27T15:56:05.1689236Z OK (skipped=1)
2022-09-27T15:56:05.1689379Z 
2022-09-27T15:56:05.1689504Z Generating XML reports...
2022-09-27T15:56:05.1722488Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155605.xml
2022-09-27T15:56:05.4760067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:05.4896189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:05.4897065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:05.4962749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:05.4969612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:05.5912017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:06.3837243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:06.3837742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:06.3839497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:06.3840003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:06.9096480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:06.9120295Z 
2022-09-27T15:56:06.9120780Z Running tests...
2022-09-27T15:56:06.9121487Z ----------------------------------------------------------------------
2022-09-27T15:56:07.1975176Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:07.1976073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:07.1977415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:07.1978183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:07.7067521Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:07.7090580Z 
2022-09-27T15:56:07.7090991Z Running tests...
2022-09-27T15:56:07.7091716Z ----------------------------------------------------------------------
2022-09-27T15:56:07.7109596Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:56:07.7110160Z 
2022-09-27T15:56:07.7110602Z ----------------------------------------------------------------------
2022-09-27T15:56:07.7111103Z Ran 1 test in 0.002s
2022-09-27T15:56:07.7111354Z 
2022-09-27T15:56:07.7111516Z OK (skipped=1)
2022-09-27T15:56:07.7111753Z 
2022-09-27T15:56:07.7111946Z Generating XML reports...
2022-09-27T15:56:07.7146807Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155607.xml
2022-09-27T15:56:08.5296838Z ok (6.935s)
2022-09-27T15:56:08.5297060Z 
2022-09-27T15:56:08.5297461Z ----------------------------------------------------------------------
2022-09-27T15:56:08.5297819Z Ran 1 test in 6.935s
2022-09-27T15:56:08.5297985Z 
2022-09-27T15:56:08.5298063Z OK
2022-09-27T15:56:08.5298202Z 
2022-09-27T15:56:08.5298577Z Generating XML reports...
2022-09-27T15:56:08.5332922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155601.xml
2022-09-27T15:56:08.5352528Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:08.5559798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20026
2022-09-27T15:56:08.5565593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20027
2022-09-27T15:56:09.7923988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:09.7924911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:09.7929961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:09.7930980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:10.2941338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:10.2942332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:10.2944702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:10.2945613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:10.3074146Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:10.3100326Z 
2022-09-27T15:56:10.3100802Z Running tests...
2022-09-27T15:56:10.3101309Z ----------------------------------------------------------------------
2022-09-27T15:56:10.3745724Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:10.3746323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:10.3747686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:10.3748197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:10.7726103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:10.7727071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:10.7728282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:10.7729237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:10.9331187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:10.9332234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:10.9333751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:10.9335076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:10.9439424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:11.0346539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:11.2749383Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:11.2771790Z 
2022-09-27T15:56:11.2772170Z Running tests...
2022-09-27T15:56:11.2772634Z ----------------------------------------------------------------------
2022-09-27T15:56:11.2790088Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:56:11.2791172Z 
2022-09-27T15:56:11.2791871Z ----------------------------------------------------------------------
2022-09-27T15:56:11.2792565Z Ran 1 test in 0.002s
2022-09-27T15:56:11.2792901Z 
2022-09-27T15:56:11.2793135Z OK (skipped=1)
2022-09-27T15:56:11.2793455Z 
2022-09-27T15:56:11.2793694Z Generating XML reports...
2022-09-27T15:56:11.2827845Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155611.xml
2022-09-27T15:56:11.9582571Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:11.9787184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20170
2022-09-27T15:56:11.9792343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20171
2022-09-27T15:56:12.5903884Z STAGE:2022-09-27 15:56:12 20027:20027 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:12.6363897Z STAGE:2022-09-27 15:56:12 20026:20026 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:12.6411552Z STAGE:2022-09-27 15:56:12 20026:20026 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:12.6412804Z STAGE:2022-09-27 15:56:12 20027:20027 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:12.6467417Z STAGE:2022-09-27 15:56:12 20026:20026 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:12.6468621Z STAGE:2022-09-27 15:56:12 20027:20027 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:13.4025570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:13.4026137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:13.4028276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:13.4028780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:13.7641875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:13.7642391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:13.7643733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:13.7644206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:13.7735563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:13.7736043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:13.7739672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:13.7740147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:13.9102571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:13.9130709Z 
2022-09-27T15:56:13.9130999Z Running tests...
2022-09-27T15:56:13.9131450Z ----------------------------------------------------------------------
2022-09-27T15:56:14.0595755Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 15:56:14 20027:20027 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:14.0597276Z STAGE:2022-09-27 15:56:14 20026:20026 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:14.0619727Z STAGE:2022-09-27 15:56:14 20026:20026 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:14.0620865Z STAGE:2022-09-27 15:56:14 20027:20027 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:14.0641592Z STAGE:2022-09-27 15:56:14 20027:20027 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:14.0643180Z STAGE:2022-09-27 15:56:14 20026:20026 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:14.2971701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:14.3001649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:14.3003051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:14.3073735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:14.3080596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:14.4018231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:15.0725670Z ok (8.160s)
2022-09-27T15:56:15.0725938Z 
2022-09-27T15:56:15.0726356Z ----------------------------------------------------------------------
2022-09-27T15:56:15.0726706Z Ran 1 test in 8.160s
2022-09-27T15:56:15.0726880Z 
2022-09-27T15:56:15.0726976Z OK
2022-09-27T15:56:15.0727094Z 
2022-09-27T15:56:15.0727510Z Generating XML reports...
2022-09-27T15:56:15.0781671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155606.xml
2022-09-27T15:56:15.6967169Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:15.7168722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20282
2022-09-27T15:56:15.7173772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20283
2022-09-27T15:56:16.8957672Z [1664294176.895206] [28fd6574525b:20170:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:56:16.8958215Z [1664294176.895206] [28fd6574525b:20170:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:56:16.8980694Z [1664294176.897707] [28fd6574525b:20171:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:56:16.8981177Z [1664294176.897707] [28fd6574525b:20171:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:56:17.2379912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:56:17.2380515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:56:17.2381300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:56:17.2382007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:56:17.4583792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:17.4584363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:17.4585520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:17.4586039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:17.4631126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:17.4631600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:17.4635146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:17.4635646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:17.4699686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:17.4700441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:17.4703416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:17.4703926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:17.9923271Z ok (7.682s)
2022-09-27T15:56:17.9923696Z 
2022-09-27T15:56:17.9924102Z ----------------------------------------------------------------------
2022-09-27T15:56:17.9924454Z Ran 1 test in 7.682s
2022-09-27T15:56:17.9924623Z 
2022-09-27T15:56:17.9924719Z OK
2022-09-27T15:56:17.9924856Z 
2022-09-27T15:56:17.9924974Z Generating XML reports...
2022-09-27T15:56:17.9949142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:17.9960285Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155610.xml
2022-09-27T15:56:18.0058487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:18.0061172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:18.0062334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:18.0084395Z 
2022-09-27T15:56:18.0084623Z Running tests...
2022-09-27T15:56:18.0085221Z ----------------------------------------------------------------------
2022-09-27T15:56:18.0153822Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:18.0160954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:18.0165194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:56:18.1077043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:18.1080306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:56:18.1081350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:56:18.1181363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:56:19.7227489Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:19.7437630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20449
2022-09-27T15:56:19.7443099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20450
2022-09-27T15:56:20.1095877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:20.1096584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:20.1098197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:20.1098693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:20.6957290Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:20.6984360Z 
2022-09-27T15:56:20.6984716Z Running tests...
2022-09-27T15:56:20.6985189Z ----------------------------------------------------------------------
2022-09-27T15:56:20.7008530Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T15:56:20.7008874Z 
2022-09-27T15:56:20.7009203Z ----------------------------------------------------------------------
2022-09-27T15:56:20.7009751Z Ran 1 test in 0.002s
2022-09-27T15:56:20.7009928Z 
2022-09-27T15:56:20.7010052Z OK (skipped=1)
2022-09-27T15:56:20.7010213Z 
2022-09-27T15:56:20.7010343Z Generating XML reports...
2022-09-27T15:56:20.7053785Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155620.xml
2022-09-27T15:56:21.2302148Z ok (7.317s)
2022-09-27T15:56:21.2302382Z 
2022-09-27T15:56:21.2302991Z ----------------------------------------------------------------------
2022-09-27T15:56:21.2303527Z Ran 1 test in 7.317s
2022-09-27T15:56:21.2303698Z 
2022-09-27T15:56:21.2303803Z OK
2022-09-27T15:56:21.2303940Z 
2022-09-27T15:56:21.2304077Z Generating XML reports...
2022-09-27T15:56:21.2339609Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155613.xml
2022-09-27T15:56:21.4102410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:21.4102920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:21.4105716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:21.4106218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:21.4199004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:21.4199783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:21.4203015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:21.4203514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:22.0092154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:22.0092738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:22.0093580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:22.0094280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:22.0099616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:22.0100120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:22.8355131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:22.8356520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:22.8357960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:22.8358899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:23.3471490Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:23.3495149Z 
2022-09-27T15:56:23.3495486Z Running tests...
2022-09-27T15:56:23.3496360Z ----------------------------------------------------------------------
2022-09-27T15:56:23.3515749Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T15:56:23.3516398Z 
2022-09-27T15:56:23.3516930Z ----------------------------------------------------------------------
2022-09-27T15:56:23.3517550Z Ran 1 test in 0.002s
2022-09-27T15:56:23.3517870Z 
2022-09-27T15:56:23.3518076Z OK (skipped=1)
2022-09-27T15:56:23.3518383Z 
2022-09-27T15:56:23.3518616Z Generating XML reports...
2022-09-27T15:56:23.3553171Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155623.xml
2022-09-27T15:56:23.4202202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:23.4203721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:23.4204944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:23.4205871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:23.6259110Z STAGE:2022-09-27 15:56:23 20449:20449 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:23.6527816Z STAGE:2022-09-27 15:56:23 20450:20450 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:23.6576198Z STAGE:2022-09-27 15:56:23 20450:20450 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:23.6577400Z STAGE:2022-09-27 15:56:23 20449:20449 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:23.6630882Z STAGE:2022-09-27 15:56:23 20450:20450 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:23.6635115Z STAGE:2022-09-27 15:56:23 20449:20449 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:23.9409310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:23.9430779Z 
2022-09-27T15:56:23.9431265Z Running tests...
2022-09-27T15:56:23.9431723Z ----------------------------------------------------------------------
2022-09-27T15:56:23.9449766Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:56:23.9450103Z 
2022-09-27T15:56:23.9450389Z ----------------------------------------------------------------------
2022-09-27T15:56:23.9450731Z Ran 1 test in 0.002s
2022-09-27T15:56:23.9450901Z 
2022-09-27T15:56:23.9451014Z OK (skipped=1)
2022-09-27T15:56:23.9451174Z 
2022-09-27T15:56:23.9451282Z Generating XML reports...
2022-09-27T15:56:23.9485298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155623.xml
2022-09-27T15:56:25.0998885Z STAGE:2022-09-27 15:56:25 20450:20450 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:25.1020911Z STAGE:2022-09-27 15:56:25 20450:20450 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:25.1238308Z STAGE:2022-09-27 15:56:25 20449:20449 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:25.1260705Z STAGE:2022-09-27 15:56:25 20449:20449 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:25.1284192Z STAGE:2022-09-27 15:56:25 20449:20449 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:25.1285431Z STAGE:2022-09-27 15:56:25 20450:20450 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:25.4585137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:25.4586069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:25.4587260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:25.4587794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:25.9617759Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:25.9641073Z 
2022-09-27T15:56:25.9641507Z Running tests...
2022-09-27T15:56:25.9642009Z ----------------------------------------------------------------------
2022-09-27T15:56:25.9664401Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-09-27T15:56:25.9664713Z 
2022-09-27T15:56:25.9665001Z ----------------------------------------------------------------------
2022-09-27T15:56:25.9665324Z Ran 1 test in 0.002s
2022-09-27T15:56:25.9665497Z 
2022-09-27T15:56:25.9665610Z OK (skipped=1)
2022-09-27T15:56:25.9665767Z 
2022-09-27T15:56:25.9665893Z Generating XML reports...
2022-09-27T15:56:25.9709854Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155625.xml
2022-09-27T15:56:25.9929198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:25.9929732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:25.9932027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:25.9932511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:26.0578993Z ok (8.049s)
2022-09-27T15:56:26.0579226Z 
2022-09-27T15:56:26.0579618Z ----------------------------------------------------------------------
2022-09-27T15:56:26.0579965Z Ran 1 test in 8.049s
2022-09-27T15:56:26.0580134Z 
2022-09-27T15:56:26.0580210Z OK
2022-09-27T15:56:26.0580352Z 
2022-09-27T15:56:26.0580488Z Generating XML reports...
2022-09-27T15:56:26.0615870Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155618.xml
2022-09-27T15:56:26.5134447Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:26.5156885Z 
2022-09-27T15:56:26.5157156Z Running tests...
2022-09-27T15:56:26.5157615Z ----------------------------------------------------------------------
2022-09-27T15:56:27.9464645Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:27.9465283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:27.9466436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:27.9466913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:28.1741884Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:28.1940077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20738
2022-09-27T15:56:28.1946758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20739
2022-09-27T15:56:28.2795907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:28.2796430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:28.2797684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:28.2798154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:28.4342642Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:28.4364622Z 
2022-09-27T15:56:28.4364771Z Running tests...
2022-09-27T15:56:28.4365540Z ----------------------------------------------------------------------
2022-09-27T15:56:28.4383966Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... skip: flaky on PyTorch CI with timeout (0.002s)
2022-09-27T15:56:28.4385097Z 
2022-09-27T15:56:28.4385431Z ----------------------------------------------------------------------
2022-09-27T15:56:28.4385789Z Ran 1 test in 0.002s
2022-09-27T15:56:28.4385956Z 
2022-09-27T15:56:28.4386071Z OK (skipped=1)
2022-09-27T15:56:28.4386211Z 
2022-09-27T15:56:28.4386342Z Generating XML reports...
2022-09-27T15:56:28.4419386Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155628.xml
2022-09-27T15:56:28.7912975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:28.7936359Z 
2022-09-27T15:56:28.7936803Z Running tests...
2022-09-27T15:56:28.7937288Z ----------------------------------------------------------------------
2022-09-27T15:56:29.8363425Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:29.8364041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:29.8365570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:29.8366061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:29.8842594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:29.8843074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:29.8845616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:29.8846113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:30.3508108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:30.3830556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:30.3831346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:30.3834110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:30.3834606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:30.4005749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:30.4006488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:30.4014182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:30.4021096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:30.4242120Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:30.4451684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20845
2022-09-27T15:56:30.4456812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20846
2022-09-27T15:56:30.5021539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:30.8884362Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:30.8907085Z 
2022-09-27T15:56:30.8907341Z Running tests...
2022-09-27T15:56:30.8907780Z ----------------------------------------------------------------------
2022-09-27T15:56:30.8925247Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-09-27T15:56:30.8926098Z 
2022-09-27T15:56:30.8926522Z ----------------------------------------------------------------------
2022-09-27T15:56:30.8926875Z Ran 1 test in 0.002s
2022-09-27T15:56:30.8927041Z 
2022-09-27T15:56:30.8927152Z OK (skipped=1)
2022-09-27T15:56:30.8927309Z 
2022-09-27T15:56:30.8927445Z Generating XML reports...
2022-09-27T15:56:30.8962555Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155630.xml
2022-09-27T15:56:30.9013367Z skip: Skipped due to small world size. (4.385s)
2022-09-27T15:56:30.9013585Z 
2022-09-27T15:56:30.9013859Z ----------------------------------------------------------------------
2022-09-27T15:56:30.9014193Z Ran 1 test in 4.386s
2022-09-27T15:56:30.9014362Z 
2022-09-27T15:56:30.9014456Z OK (skipped=1)
2022-09-27T15:56:30.9014611Z 
2022-09-27T15:56:30.9014738Z Generating XML reports...
2022-09-27T15:56:30.9052128Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155626.xml
2022-09-27T15:56:32.1393933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:32.1404904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:32.1413483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:32.1415069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:32.2662785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:32.2663289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:32.2665271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:32.2665777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:32.7848007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:32.7848592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:32.7849410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:32.7850380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:32.7956068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:32.8862139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:32.9816852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:32.9817362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:32.9819597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:32.9820095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:33.1375968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:33.1376500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:33.1378687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:33.1379187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:33.5105027Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:33.5130077Z 
2022-09-27T15:56:33.5130303Z Running tests...
2022-09-27T15:56:33.5130749Z ----------------------------------------------------------------------
2022-09-27T15:56:33.6390156Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:33.6413110Z 
2022-09-27T15:56:33.6413625Z Running tests...
2022-09-27T15:56:33.6414160Z ----------------------------------------------------------------------
2022-09-27T15:56:34.4301202Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 15:56:34 20846:20846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:34.5022642Z STAGE:2022-09-27 15:56:34 20845:20845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:34.5075188Z STAGE:2022-09-27 15:56:34 20845:20845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:34.5077238Z STAGE:2022-09-27 15:56:34 20846:20846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:34.5129224Z STAGE:2022-09-27 15:56:34 20845:20845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:34.5135299Z STAGE:2022-09-27 15:56:34 20846:20846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:35.1677107Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:35.1885862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20997
2022-09-27T15:56:35.1890651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20998
2022-09-27T15:56:35.2529979Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:35.2729715Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21000
2022-09-27T15:56:35.2734691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21001
2022-09-27T15:56:35.8614626Z STAGE:2022-09-27 15:56:35 20846:20846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:35.8618217Z STAGE:2022-09-27 15:56:35 20845:20845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:35.8644107Z STAGE:2022-09-27 15:56:35 20846:20846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:35.8657942Z STAGE:2022-09-27 15:56:35 20845:20845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:56:35.8683935Z STAGE:2022-09-27 15:56:35 20846:20846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:35.8688569Z STAGE:2022-09-27 15:56:35 20845:20845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:56:36.8593545Z ok (8.065s)
2022-09-27T15:56:36.8593900Z 
2022-09-27T15:56:36.8594307Z ----------------------------------------------------------------------
2022-09-27T15:56:36.8594636Z Ran 1 test in 8.066s
2022-09-27T15:56:36.8594805Z 
2022-09-27T15:56:36.8594901Z OK
2022-09-27T15:56:36.8595037Z 
2022-09-27T15:56:36.8595173Z Generating XML reports...
2022-09-27T15:56:36.8630351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155628.xml
2022-09-27T15:56:36.8831006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:36.8831668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:36.8833042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:36.8833648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:36.9423864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:36.9424359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:36.9425427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:36.9426090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:37.0021903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:37.0022410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:37.0025032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:37.0025672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:37.0060617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:37.0061249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:37.0064724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:37.0065435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:37.4130038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:37.4505514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:37.4506369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:37.4535487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:37.4542575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:37.5227989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:37.5472592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:37.5473368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:37.5516024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:37.5532499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:37.5539503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:37.6487820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:37.8961790Z skip: Skipped due to small world size. (4.383s)
2022-09-27T15:56:37.8962091Z 
2022-09-27T15:56:37.8962487Z ----------------------------------------------------------------------
2022-09-27T15:56:37.8962827Z Ran 1 test in 4.383s
2022-09-27T15:56:37.8962994Z 
2022-09-27T15:56:37.8963102Z OK (skipped=1)
2022-09-27T15:56:37.8963243Z 
2022-09-27T15:56:37.8963369Z Generating XML reports...
2022-09-27T15:56:37.8999186Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155633.xml
2022-09-27T15:56:39.0620941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:39.0621455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:39.0626284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:39.0626776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:39.5793028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:39.5816997Z 
2022-09-27T15:56:39.5817506Z Running tests...
2022-09-27T15:56:39.5817995Z ----------------------------------------------------------------------
2022-09-27T15:56:39.5835729Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:56:39.5836036Z 
2022-09-27T15:56:39.5836328Z ----------------------------------------------------------------------
2022-09-27T15:56:39.5836926Z Ran 1 test in 0.002s
2022-09-27T15:56:39.5837133Z 
2022-09-27T15:56:39.5837254Z OK (skipped=1)
2022-09-27T15:56:39.5837410Z 
2022-09-27T15:56:39.5837537Z Generating XML reports...
2022-09-27T15:56:39.5873288Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155639.xml
2022-09-27T15:56:40.0971641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:56:40.0976820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:56:40.0977629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:56:40.1073042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:56:40.1219709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:40.1220205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:40.1223376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:40.1223879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:40.7050364Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:40.7074210Z 
2022-09-27T15:56:40.7074598Z Running tests...
2022-09-27T15:56:40.7075050Z ----------------------------------------------------------------------
2022-09-27T15:56:40.7093957Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-09-27T15:56:40.7094395Z 
2022-09-27T15:56:40.7094790Z ----------------------------------------------------------------------
2022-09-27T15:56:40.7095113Z Ran 1 test in 0.002s
2022-09-27T15:56:40.7095277Z 
2022-09-27T15:56:40.7095404Z OK (skipped=1)
2022-09-27T15:56:40.7095559Z 
2022-09-27T15:56:40.7095683Z Generating XML reports...
2022-09-27T15:56:40.7131072Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155640.xml
2022-09-27T15:56:40.7855628Z ok (7.144s)
2022-09-27T15:56:40.7855989Z 
2022-09-27T15:56:40.7856625Z ----------------------------------------------------------------------
2022-09-27T15:56:40.7857279Z Ran 1 test in 7.144s
2022-09-27T15:56:40.7857588Z 
2022-09-27T15:56:40.7857736Z OK
2022-09-27T15:56:40.7857990Z 
2022-09-27T15:56:40.7858229Z Generating XML reports...
2022-09-27T15:56:40.7894738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155633.xml
2022-09-27T15:56:41.5267069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:41.5267607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:41.5269129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:41.5269617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:42.1254299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:42.1275923Z 
2022-09-27T15:56:42.1276185Z Running tests...
2022-09-27T15:56:42.1276638Z ----------------------------------------------------------------------
2022-09-27T15:56:42.1295071Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:56:42.1295432Z 
2022-09-27T15:56:42.1295911Z ----------------------------------------------------------------------
2022-09-27T15:56:42.1296275Z Ran 1 test in 0.002s
2022-09-27T15:56:42.1296441Z 
2022-09-27T15:56:42.1296552Z OK (skipped=1)
2022-09-27T15:56:42.1296708Z 
2022-09-27T15:56:42.1296851Z Generating XML reports...
2022-09-27T15:56:42.1332089Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155642.xml
2022-09-27T15:56:42.7064260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:42.7064781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:42.7066922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:42.7067423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:42.8515644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:42.8516146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:42.8518812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:42.8519347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:43.2659002Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:43.2682866Z 
2022-09-27T15:56:43.2683129Z Running tests...
2022-09-27T15:56:43.2683550Z ----------------------------------------------------------------------
2022-09-27T15:56:43.4005234Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:43.4027966Z 
2022-09-27T15:56:43.4028421Z Running tests...
2022-09-27T15:56:43.4028915Z ----------------------------------------------------------------------
2022-09-27T15:56:44.0516355Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:44.0516974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:44.0519201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:44.0520009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:44.5648119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:44.5672151Z 
2022-09-27T15:56:44.5672456Z Running tests...
2022-09-27T15:56:44.5672901Z ----------------------------------------------------------------------
2022-09-27T15:56:44.5692693Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:56:44.5693589Z 
2022-09-27T15:56:44.5693903Z ----------------------------------------------------------------------
2022-09-27T15:56:44.5694425Z Ran 1 test in 0.002s
2022-09-27T15:56:44.5694724Z 
2022-09-27T15:56:44.5694849Z OK (skipped=1)
2022-09-27T15:56:44.5695025Z 
2022-09-27T15:56:44.5695138Z Generating XML reports...
2022-09-27T15:56:44.5730802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155644.xml
2022-09-27T15:56:44.8685117Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:44.8889450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21361
2022-09-27T15:56:44.8894331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21362
2022-09-27T15:56:44.9926711Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:45.0124267Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21364
2022-09-27T15:56:45.0130226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21365
2022-09-27T15:56:46.5977540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.5978109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.5979071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.5979610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.6001731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.6002210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.6005885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.6006370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.6013997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.6014466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.6018144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.6018654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.7824970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.7825479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.7827783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.7828274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:46.8315654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:46.8316137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:46.8319235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:46.8319725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:47.1013775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:47.1036857Z 
2022-09-27T15:56:47.1037306Z Running tests...
2022-09-27T15:56:47.1037818Z ----------------------------------------------------------------------
2022-09-27T15:56:47.1056995Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:56:47.1057325Z 
2022-09-27T15:56:47.1057617Z ----------------------------------------------------------------------
2022-09-27T15:56:47.1057956Z Ran 1 test in 0.002s
2022-09-27T15:56:47.1058124Z 
2022-09-27T15:56:47.1058232Z OK (skipped=1)
2022-09-27T15:56:47.1058373Z 
2022-09-27T15:56:47.1058499Z Generating XML reports...
2022-09-27T15:56:47.1093827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155647.xml
2022-09-27T15:56:47.1164034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:47.1189310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:47.1190023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:47.1267612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:47.1274377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:47.2247619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:47.2927103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:47.3472366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:47.3473234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:47.3538350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:47.3545199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:47.4485132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:47.5963497Z skip: Skipped due to small world size. (4.328s)
2022-09-27T15:56:47.5963942Z 
2022-09-27T15:56:47.5964352Z ----------------------------------------------------------------------
2022-09-27T15:56:47.5964674Z Ran 1 test in 4.328s
2022-09-27T15:56:47.5964837Z 
2022-09-27T15:56:47.5964946Z OK (skipped=1)
2022-09-27T15:56:47.5965103Z 
2022-09-27T15:56:47.5965514Z Generating XML reports...
2022-09-27T15:56:47.5999619Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155643.xml
2022-09-27T15:56:47.8199238Z skip: Need at least 3 CUDA devices (4.417s)
2022-09-27T15:56:47.8199705Z 
2022-09-27T15:56:47.8200801Z ----------------------------------------------------------------------
2022-09-27T15:56:47.8201195Z Ran 1 test in 4.417s
2022-09-27T15:56:47.8201361Z 
2022-09-27T15:56:47.8201471Z OK (skipped=1)
2022-09-27T15:56:47.8201631Z 
2022-09-27T15:56:47.8201760Z Generating XML reports...
2022-09-27T15:56:47.8236492Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155643.xml
2022-09-27T15:56:49.2436230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:49.2436761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:49.2438118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:49.2438609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:49.7205882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:49.7206383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:49.7207645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:49.7208120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:49.7448917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:49.7473066Z 
2022-09-27T15:56:49.7473421Z Running tests...
2022-09-27T15:56:49.7473907Z ----------------------------------------------------------------------
2022-09-27T15:56:49.7492133Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:56:49.7492461Z 
2022-09-27T15:56:49.7492754Z ----------------------------------------------------------------------
2022-09-27T15:56:49.7493084Z Ran 1 test in 0.002s
2022-09-27T15:56:49.7493250Z 
2022-09-27T15:56:49.7493358Z OK (skipped=1)
2022-09-27T15:56:49.7493496Z 
2022-09-27T15:56:49.7493620Z Generating XML reports...
2022-09-27T15:56:49.7528239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155649.xml
2022-09-27T15:56:49.9151014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:49.9151521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:49.9154032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:49.9154570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:50.2144643Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:50.2170290Z 
2022-09-27T15:56:50.2170577Z Running tests...
2022-09-27T15:56:50.2171021Z ----------------------------------------------------------------------
2022-09-27T15:56:50.2189438Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T15:56:50.2190079Z 
2022-09-27T15:56:50.2190376Z ----------------------------------------------------------------------
2022-09-27T15:56:50.2190700Z Ran 1 test in 0.002s
2022-09-27T15:56:50.2190863Z 
2022-09-27T15:56:50.2190973Z OK (skipped=1)
2022-09-27T15:56:50.2191127Z 
2022-09-27T15:56:50.2191253Z Generating XML reports...
2022-09-27T15:56:50.2226040Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155650.xml
2022-09-27T15:56:50.4111464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:50.4134273Z 
2022-09-27T15:56:50.4134783Z Running tests...
2022-09-27T15:56:50.4135245Z ----------------------------------------------------------------------
2022-09-27T15:56:50.4153902Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-09-27T15:56:50.4154414Z 
2022-09-27T15:56:50.4154825Z ----------------------------------------------------------------------
2022-09-27T15:56:50.4155198Z Ran 1 test in 0.002s
2022-09-27T15:56:50.4155412Z 
2022-09-27T15:56:50.4155610Z OK (skipped=1)
2022-09-27T15:56:50.4155874Z 
2022-09-27T15:56:50.4156005Z Generating XML reports...
2022-09-27T15:56:50.4190443Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155650.xml
2022-09-27T15:56:51.7891882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:51.7892876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:51.7894507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:51.7895451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:52.1631029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:52.1631975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:52.1633146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:52.1634110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:52.2869370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:52.2892255Z 
2022-09-27T15:56:52.2892729Z Running tests...
2022-09-27T15:56:52.2893554Z ----------------------------------------------------------------------
2022-09-27T15:56:52.2911798Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T15:56:52.2912443Z 
2022-09-27T15:56:52.2912985Z ----------------------------------------------------------------------
2022-09-27T15:56:52.2913622Z Ran 1 test in 0.002s
2022-09-27T15:56:52.2913927Z 
2022-09-27T15:56:52.2914111Z OK (skipped=1)
2022-09-27T15:56:52.2914410Z 
2022-09-27T15:56:52.2914658Z Generating XML reports...
2022-09-27T15:56:52.2947789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155652.xml
2022-09-27T15:56:52.4389940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:52.4390479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:52.4392718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:52.4393226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:52.6566419Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:52.6588859Z 
2022-09-27T15:56:52.6589158Z Running tests...
2022-09-27T15:56:52.6589608Z ----------------------------------------------------------------------
2022-09-27T15:56:52.6639435Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.005s)
2022-09-27T15:56:52.6639760Z 
2022-09-27T15:56:52.6640482Z ----------------------------------------------------------------------
2022-09-27T15:56:52.6640834Z Ran 1 test in 0.005s
2022-09-27T15:56:52.6640999Z 
2022-09-27T15:56:52.6641119Z OK (skipped=1)
2022-09-27T15:56:52.6641278Z 
2022-09-27T15:56:52.6641659Z Generating XML reports...
2022-09-27T15:56:52.6675701Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155652.xml
2022-09-27T15:56:52.9330229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:52.9351039Z 
2022-09-27T15:56:52.9351320Z Running tests...
2022-09-27T15:56:52.9351763Z ----------------------------------------------------------------------
2022-09-27T15:56:52.9370069Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-09-27T15:56:52.9370389Z 
2022-09-27T15:56:52.9370677Z ----------------------------------------------------------------------
2022-09-27T15:56:52.9371009Z Ran 1 test in 0.002s
2022-09-27T15:56:52.9371175Z 
2022-09-27T15:56:52.9371293Z OK (skipped=1)
2022-09-27T15:56:52.9371455Z 
2022-09-27T15:56:52.9371567Z Generating XML reports...
2022-09-27T15:56:52.9406538Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155652.xml
2022-09-27T15:56:54.3161417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:54.3162755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:54.3164008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:54.3164507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:54.6801649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:54.6802204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:54.6804762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:54.6805591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:54.8090034Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:54.8114185Z 
2022-09-27T15:56:54.8114494Z Running tests...
2022-09-27T15:56:54.8115376Z ----------------------------------------------------------------------
2022-09-27T15:56:54.8133815Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T15:56:54.8134257Z 
2022-09-27T15:56:54.8135273Z ----------------------------------------------------------------------
2022-09-27T15:56:54.8136008Z Ran 1 test in 0.002s
2022-09-27T15:56:54.8136324Z 
2022-09-27T15:56:54.8136542Z OK (skipped=1)
2022-09-27T15:56:54.8136837Z 
2022-09-27T15:56:54.8137076Z Generating XML reports...
2022-09-27T15:56:54.8173629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155654.xml
2022-09-27T15:56:54.9714137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:54.9715050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:54.9716205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:54.9717074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:55.1728253Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:55.1751633Z 
2022-09-27T15:56:55.1752146Z Running tests...
2022-09-27T15:56:55.1752635Z ----------------------------------------------------------------------
2022-09-27T15:56:55.1771203Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T15:56:55.1771532Z 
2022-09-27T15:56:55.1772124Z ----------------------------------------------------------------------
2022-09-27T15:56:55.1772478Z Ran 1 test in 0.002s
2022-09-27T15:56:55.1772626Z 
2022-09-27T15:56:55.1772743Z OK (skipped=1)
2022-09-27T15:56:55.1772898Z 
2022-09-27T15:56:55.1773025Z Generating XML reports...
2022-09-27T15:56:55.1806471Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155655.xml
2022-09-27T15:56:55.4719967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:56:55.4742849Z 
2022-09-27T15:56:55.4743189Z Running tests...
2022-09-27T15:56:55.4744447Z ----------------------------------------------------------------------
2022-09-27T15:56:56.7280142Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:56.7281652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:56.7282880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:56.7283878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:57.0919490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:57.0920811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:57.0921991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:57.0922937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:57.1198959Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:56:57.1398339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21899
2022-09-27T15:56:57.1405279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21900
2022-09-27T15:56:57.2153792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:57.2176188Z 
2022-09-27T15:56:57.2176541Z Running tests...
2022-09-27T15:56:57.2176985Z ----------------------------------------------------------------------
2022-09-27T15:56:57.2195580Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:56:57.2195908Z 
2022-09-27T15:56:57.2196189Z ----------------------------------------------------------------------
2022-09-27T15:56:57.2196502Z Ran 1 test in 0.002s
2022-09-27T15:56:57.2196667Z 
2022-09-27T15:56:57.2196772Z OK (skipped=1)
2022-09-27T15:56:57.2196932Z 
2022-09-27T15:56:57.2197057Z Generating XML reports...
2022-09-27T15:56:57.2230827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155657.xml
2022-09-27T15:56:57.5830524Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:56:57.5853160Z 
2022-09-27T15:56:57.5853423Z Running tests...
2022-09-27T15:56:57.5853877Z ----------------------------------------------------------------------
2022-09-27T15:56:57.5877028Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-09-27T15:56:57.5877353Z 
2022-09-27T15:56:57.5877631Z ----------------------------------------------------------------------
2022-09-27T15:56:57.5877951Z Ran 1 test in 0.002s
2022-09-27T15:56:57.5878117Z 
2022-09-27T15:56:57.5878228Z OK (skipped=1)
2022-09-27T15:56:57.5878385Z 
2022-09-27T15:56:57.5878514Z Generating XML reports...
2022-09-27T15:56:57.5913571Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155657.xml
2022-09-27T15:56:58.8640454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:58.8641458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:58.8642094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:58.8642587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:58.8933795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:58.8934288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:58.8936844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:58.8937349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:59.2569860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:59.2570499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:59.2572444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:59.2572948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:59.3667083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:56:59.3999195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:56:59.3999980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:59.4072497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:56:59.4078873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:56:59.5012748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:56:59.6035783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:56:59.6036407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:56:59.6038478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:56:59.6038966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:56:59.7639905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:56:59.7663386Z 
2022-09-27T15:56:59.7663841Z Running tests...
2022-09-27T15:56:59.7664349Z ----------------------------------------------------------------------
2022-09-27T15:56:59.7683254Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:56:59.7683621Z 
2022-09-27T15:56:59.7683974Z ----------------------------------------------------------------------
2022-09-27T15:56:59.7684338Z Ran 1 test in 0.002s
2022-09-27T15:56:59.7684489Z 
2022-09-27T15:56:59.7684604Z OK (skipped=1)
2022-09-27T15:56:59.7684774Z 
2022-09-27T15:56:59.7684901Z Generating XML reports...
2022-09-27T15:56:59.7719196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155659.xml
2022-09-27T15:57:00.1105575Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:00.1128511Z 
2022-09-27T15:57:00.1128659Z Running tests...
2022-09-27T15:57:00.1129427Z ----------------------------------------------------------------------
2022-09-27T15:57:00.1152335Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-09-27T15:57:00.1152659Z 
2022-09-27T15:57:00.1152983Z ----------------------------------------------------------------------
2022-09-27T15:57:00.1153320Z Ran 1 test in 0.002s
2022-09-27T15:57:00.1153821Z 
2022-09-27T15:57:00.1153960Z OK (skipped=1)
2022-09-27T15:57:00.1154124Z 
2022-09-27T15:57:00.1154253Z Generating XML reports...
2022-09-27T15:57:00.1189271Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155700.xml
2022-09-27T15:57:01.8252587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:01.8253090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:01.8255341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:01.8255842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:02.2012859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:02.2013370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:02.2015544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:02.2016261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:02.3314493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:02.3338629Z 
2022-09-27T15:57:02.3338886Z Running tests...
2022-09-27T15:57:02.3339322Z ----------------------------------------------------------------------
2022-09-27T15:57:02.3357309Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:57:02.3358547Z 
2022-09-27T15:57:02.3358972Z ----------------------------------------------------------------------
2022-09-27T15:57:02.3359310Z Ran 1 test in 0.002s
2022-09-27T15:57:02.3359482Z 
2022-09-27T15:57:02.3359595Z OK (skipped=1)
2022-09-27T15:57:02.3359753Z 
2022-09-27T15:57:02.3359878Z Generating XML reports...
2022-09-27T15:57:02.3395464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155702.xml
2022-09-27T15:57:02.6819743Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:02.6841579Z 
2022-09-27T15:57:02.6841725Z Running tests...
2022-09-27T15:57:02.6842528Z ----------------------------------------------------------------------
2022-09-27T15:57:02.6865880Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:57:02.6866214Z 
2022-09-27T15:57:02.6866502Z ----------------------------------------------------------------------
2022-09-27T15:57:02.6866844Z Ran 1 test in 0.002s
2022-09-27T15:57:02.6866993Z 
2022-09-27T15:57:02.6867107Z OK (skipped=1)
2022-09-27T15:57:02.6867270Z 
2022-09-27T15:57:02.6867397Z Generating XML reports...
2022-09-27T15:57:02.6901434Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155702.xml
2022-09-27T15:57:03.3539702Z ok (7.879s)
2022-09-27T15:57:03.3539945Z 
2022-09-27T15:57:03.3540335Z ----------------------------------------------------------------------
2022-09-27T15:57:03.3540684Z Ran 1 test in 7.880s
2022-09-27T15:57:03.3540854Z 
2022-09-27T15:57:03.3540954Z OK
2022-09-27T15:57:03.3541092Z 
2022-09-27T15:57:03.3541209Z Generating XML reports...
2022-09-27T15:57:03.3576552Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155655.xml
2022-09-27T15:57:04.2874215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:04.2874716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:04.2876630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:04.2877141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:04.7113764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:04.7114280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:04.7116292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:04.7116786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:04.8478588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:04.8504063Z 
2022-09-27T15:57:04.8504284Z Running tests...
2022-09-27T15:57:04.8504714Z ----------------------------------------------------------------------
2022-09-27T15:57:04.8525421Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:57:04.8526243Z 
2022-09-27T15:57:04.8526692Z ----------------------------------------------------------------------
2022-09-27T15:57:04.8527344Z Ran 1 test in 0.002s
2022-09-27T15:57:04.8527514Z 
2022-09-27T15:57:04.8527619Z OK (skipped=1)
2022-09-27T15:57:04.8527787Z 
2022-09-27T15:57:04.8527919Z Generating XML reports...
2022-09-27T15:57:04.8563523Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155704.xml
2022-09-27T15:57:05.1963845Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:05.1986083Z 
2022-09-27T15:57:05.1986592Z Running tests...
2022-09-27T15:57:05.1987104Z ----------------------------------------------------------------------
2022-09-27T15:57:05.2015370Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T15:57:05.2015678Z 
2022-09-27T15:57:05.2015975Z ----------------------------------------------------------------------
2022-09-27T15:57:05.2016316Z Ran 1 test in 0.003s
2022-09-27T15:57:05.2016487Z 
2022-09-27T15:57:05.2016607Z OK (skipped=1)
2022-09-27T15:57:05.2016771Z 
2022-09-27T15:57:05.2016879Z Generating XML reports...
2022-09-27T15:57:05.2050959Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155705.xml
2022-09-27T15:57:05.4839914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:05.4841061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:05.4843120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:05.4843612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:05.9685255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:05.9707477Z 
2022-09-27T15:57:05.9707876Z Running tests...
2022-09-27T15:57:05.9708360Z ----------------------------------------------------------------------
2022-09-27T15:57:05.9727281Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-09-27T15:57:05.9727604Z 
2022-09-27T15:57:05.9727876Z ----------------------------------------------------------------------
2022-09-27T15:57:05.9728209Z Ran 1 test in 0.002s
2022-09-27T15:57:05.9728368Z 
2022-09-27T15:57:05.9728479Z OK (skipped=1)
2022-09-27T15:57:05.9728637Z 
2022-09-27T15:57:05.9728764Z Generating XML reports...
2022-09-27T15:57:05.9763724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155705.xml
2022-09-27T15:57:06.7423230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:06.7424637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:06.7425861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:06.7426841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:07.1719533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:07.1720015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:07.1721826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:07.1722317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:07.3315430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:07.3338501Z 
2022-09-27T15:57:07.3338974Z Running tests...
2022-09-27T15:57:07.3339510Z ----------------------------------------------------------------------
2022-09-27T15:57:07.3357945Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:57:07.3358636Z 
2022-09-27T15:57:07.3358908Z ----------------------------------------------------------------------
2022-09-27T15:57:07.3359236Z Ran 1 test in 0.002s
2022-09-27T15:57:07.3359401Z 
2022-09-27T15:57:07.3359508Z OK (skipped=1)
2022-09-27T15:57:07.3359664Z 
2022-09-27T15:57:07.3359789Z Generating XML reports...
2022-09-27T15:57:07.3393742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155707.xml
2022-09-27T15:57:07.6571351Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:07.6593634Z 
2022-09-27T15:57:07.6594062Z Running tests...
2022-09-27T15:57:07.6594603Z ----------------------------------------------------------------------
2022-09-27T15:57:07.6620736Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T15:57:07.6621388Z 
2022-09-27T15:57:07.6621851Z ----------------------------------------------------------------------
2022-09-27T15:57:07.6622191Z Ran 1 test in 0.003s
2022-09-27T15:57:07.6622364Z 
2022-09-27T15:57:07.6622458Z OK (skipped=1)
2022-09-27T15:57:07.6622613Z 
2022-09-27T15:57:07.6622740Z Generating XML reports...
2022-09-27T15:57:07.6656294Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155707.xml
2022-09-27T15:57:07.9357207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:07.9357729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:07.9359763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:07.9360485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:08.4974703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:08.4997406Z 
2022-09-27T15:57:08.4997732Z Running tests...
2022-09-27T15:57:08.4998162Z ----------------------------------------------------------------------
2022-09-27T15:57:09.2582512Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:09.2583093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:09.2584473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:09.2584965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:09.6670743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:09.6671243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:09.6672807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:09.6673458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:09.7584847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:09.7608346Z 
2022-09-27T15:57:09.7608786Z Running tests...
2022-09-27T15:57:09.7609263Z ----------------------------------------------------------------------
2022-09-27T15:57:09.7628098Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:57:09.7628725Z 
2022-09-27T15:57:09.7629208Z ----------------------------------------------------------------------
2022-09-27T15:57:09.7629555Z Ran 1 test in 0.002s
2022-09-27T15:57:09.7629705Z 
2022-09-27T15:57:09.7629826Z OK (skipped=1)
2022-09-27T15:57:09.7629983Z 
2022-09-27T15:57:09.7630426Z Generating XML reports...
2022-09-27T15:57:09.7665705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155709.xml
2022-09-27T15:57:10.0983182Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:10.1175356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22380
2022-09-27T15:57:10.1181219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22381
2022-09-27T15:57:10.1792027Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:10.1816213Z 
2022-09-27T15:57:10.1816499Z Running tests...
2022-09-27T15:57:10.1816949Z ----------------------------------------------------------------------
2022-09-27T15:57:10.1838303Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:57:10.1838890Z 
2022-09-27T15:57:10.1839226Z ----------------------------------------------------------------------
2022-09-27T15:57:10.1839577Z Ran 1 test in 0.002s
2022-09-27T15:57:10.1839745Z 
2022-09-27T15:57:10.1839858Z OK (skipped=1)
2022-09-27T15:57:10.1840017Z 
2022-09-27T15:57:10.1840131Z Generating XML reports...
2022-09-27T15:57:10.1875320Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155710.xml
2022-09-27T15:57:11.7501711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:11.7502243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:11.7503641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:11.7504132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:11.7581773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:11.7582262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:11.7585607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:11.7586076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:11.7840850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:11.7841597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:11.7844849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:11.7845320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:12.2013374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:12.2013884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:12.2015599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:12.2016100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:12.2623900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:12.2681066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:12.2706077Z 
2022-09-27T15:57:12.2706544Z Running tests...
2022-09-27T15:57:12.2707090Z ----------------------------------------------------------------------
2022-09-27T15:57:12.2726439Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:57:12.2726812Z 
2022-09-27T15:57:12.2727101Z ----------------------------------------------------------------------
2022-09-27T15:57:12.2727681Z Ran 1 test in 0.002s
2022-09-27T15:57:12.2727850Z 
2022-09-27T15:57:12.2727963Z OK (skipped=1)
2022-09-27T15:57:12.2728124Z 
2022-09-27T15:57:12.2728256Z Generating XML reports...
2022-09-27T15:57:12.2762890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155712.xml
2022-09-27T15:57:12.2926083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:12.2926872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:12.2929488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:12.2936910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:12.3939940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:12.7040643Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:12.7062423Z 
2022-09-27T15:57:12.7062630Z Running tests...
2022-09-27T15:57:12.7063052Z ----------------------------------------------------------------------
2022-09-27T15:57:12.7082617Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:57:12.7082945Z 
2022-09-27T15:57:12.7083237Z ----------------------------------------------------------------------
2022-09-27T15:57:12.7083574Z Ran 1 test in 0.002s
2022-09-27T15:57:12.7083744Z 
2022-09-27T15:57:12.7083841Z OK (skipped=1)
2022-09-27T15:57:12.7084003Z 
2022-09-27T15:57:12.7084132Z Generating XML reports...
2022-09-27T15:57:12.7117190Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155712.xml
2022-09-27T15:57:12.7243898Z skip: Skipped due to small world size. (4.224s)
2022-09-27T15:57:12.7244166Z 
2022-09-27T15:57:12.7244482Z ----------------------------------------------------------------------
2022-09-27T15:57:12.7244822Z Ran 1 test in 4.225s
2022-09-27T15:57:12.7244991Z 
2022-09-27T15:57:12.7245101Z OK (skipped=1)
2022-09-27T15:57:12.7245258Z 
2022-09-27T15:57:12.7245396Z Generating XML reports...
2022-09-27T15:57:12.7282126Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155708.xml
2022-09-27T15:57:14.4084469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:14.4084985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:14.4086723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:14.4087241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:14.7811702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:14.7812226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:14.7813417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:14.7813891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:14.8906115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:14.8928395Z 
2022-09-27T15:57:14.8928636Z Running tests...
2022-09-27T15:57:14.8929093Z ----------------------------------------------------------------------
2022-09-27T15:57:14.8946475Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:57:14.8946829Z 
2022-09-27T15:57:14.8947114Z ----------------------------------------------------------------------
2022-09-27T15:57:14.8947764Z Ran 1 test in 0.002s
2022-09-27T15:57:14.8947932Z 
2022-09-27T15:57:14.8948045Z OK (skipped=1)
2022-09-27T15:57:14.8948203Z 
2022-09-27T15:57:14.8948332Z Generating XML reports...
2022-09-27T15:57:14.8982801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155714.xml
2022-09-27T15:57:14.9594234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:14.9594736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:14.9596995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:14.9597512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:15.2653156Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:15.2675346Z 
2022-09-27T15:57:15.2675834Z Running tests...
2022-09-27T15:57:15.2676342Z ----------------------------------------------------------------------
2022-09-27T15:57:15.2700543Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:57:15.2700893Z 
2022-09-27T15:57:15.2701181Z ----------------------------------------------------------------------
2022-09-27T15:57:15.2701520Z Ran 1 test in 0.002s
2022-09-27T15:57:15.2701686Z 
2022-09-27T15:57:15.2701799Z OK (skipped=1)
2022-09-27T15:57:15.2701955Z 
2022-09-27T15:57:15.2702082Z Generating XML reports...
2022-09-27T15:57:15.2735904Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155715.xml
2022-09-27T15:57:15.4751134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:15.4774874Z 
2022-09-27T15:57:15.4775290Z Running tests...
2022-09-27T15:57:15.4775748Z ----------------------------------------------------------------------
2022-09-27T15:57:15.4794417Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-09-27T15:57:15.4794987Z 
2022-09-27T15:57:15.4795644Z ----------------------------------------------------------------------
2022-09-27T15:57:15.4796200Z Ran 1 test in 0.002s
2022-09-27T15:57:15.4796367Z 
2022-09-27T15:57:15.4796461Z OK (skipped=1)
2022-09-27T15:57:15.4796615Z 
2022-09-27T15:57:15.4796741Z Generating XML reports...
2022-09-27T15:57:15.4831262Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155715.xml
2022-09-27T15:57:16.9379932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:16.9380537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:16.9384531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:16.9385047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:17.2588630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:17.2589138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:17.2590891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:17.2591365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:17.4588998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:17.4611008Z 
2022-09-27T15:57:17.4611487Z Running tests...
2022-09-27T15:57:17.4612019Z ----------------------------------------------------------------------
2022-09-27T15:57:17.4631210Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:57:17.4631961Z 
2022-09-27T15:57:17.4632289Z ----------------------------------------------------------------------
2022-09-27T15:57:17.4632618Z Ran 1 test in 0.002s
2022-09-27T15:57:17.4632785Z 
2022-09-27T15:57:17.4632895Z OK (skipped=1)
2022-09-27T15:57:17.4633054Z 
2022-09-27T15:57:17.4633181Z Generating XML reports...
2022-09-27T15:57:17.4668241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155717.xml
2022-09-27T15:57:17.5562302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:17.5562809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:17.5565515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:17.5566017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:17.7488423Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:17.7510620Z 
2022-09-27T15:57:17.7510836Z Running tests...
2022-09-27T15:57:17.7511263Z ----------------------------------------------------------------------
2022-09-27T15:57:17.7536253Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:57:17.7536699Z 
2022-09-27T15:57:17.7537319Z ----------------------------------------------------------------------
2022-09-27T15:57:17.7537766Z Ran 1 test in 0.003s
2022-09-27T15:57:17.7537922Z 
2022-09-27T15:57:17.7538040Z OK (skipped=1)
2022-09-27T15:57:17.7538201Z 
2022-09-27T15:57:17.7538435Z Generating XML reports...
2022-09-27T15:57:17.7572925Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155717.xml
2022-09-27T15:57:18.0710659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:18.0733602Z 
2022-09-27T15:57:18.0734089Z Running tests...
2022-09-27T15:57:19.3954158Z ----------------------------------------------------------------------
2022-09-27T15:57:19.3955748Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:19.3956816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:19.3958066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:19.3959026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:19.6666055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:19.6667068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:19.6668348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:19.6668877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:19.7084312Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:19.7283940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22783
2022-09-27T15:57:19.7290182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22784
2022-09-27T15:57:19.8886358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:19.8909395Z 
2022-09-27T15:57:19.8909695Z Running tests...
2022-09-27T15:57:19.8910147Z ----------------------------------------------------------------------
2022-09-27T15:57:19.8928223Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:57:19.8929214Z 
2022-09-27T15:57:19.8929709Z ----------------------------------------------------------------------
2022-09-27T15:57:19.8930047Z Ran 1 test in 0.002s
2022-09-27T15:57:19.8930217Z 
2022-09-27T15:57:19.8930309Z OK (skipped=1)
2022-09-27T15:57:19.8930474Z 
2022-09-27T15:57:19.8930603Z Generating XML reports...
2022-09-27T15:57:19.8964660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155719.xml
2022-09-27T15:57:20.1626223Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:20.1649639Z 
2022-09-27T15:57:20.1649859Z Running tests...
2022-09-27T15:57:20.1650321Z ----------------------------------------------------------------------
2022-09-27T15:57:20.1672303Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:57:20.1672729Z 
2022-09-27T15:57:20.1673288Z ----------------------------------------------------------------------
2022-09-27T15:57:20.1673661Z Ran 1 test in 0.002s
2022-09-27T15:57:20.1673836Z 
2022-09-27T15:57:20.1673947Z OK (skipped=1)
2022-09-27T15:57:20.1674105Z 
2022-09-27T15:57:20.1674238Z Generating XML reports...
2022-09-27T15:57:20.1709210Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155720.xml
2022-09-27T15:57:21.5069131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:21.5069646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:21.5070738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:21.5071243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:21.5526450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:21.5526974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:21.5529087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:21.5529582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:21.8800747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:21.8801270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:21.8802755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:21.8803534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:22.0193283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:22.0615050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:22.0615888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:22.0700201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:22.0707372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:22.1629157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:22.2659911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:22.2660438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:22.2662394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:22.2663183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:22.3792447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:22.3814829Z 
2022-09-27T15:57:22.3815194Z Running tests...
2022-09-27T15:57:22.3815649Z ----------------------------------------------------------------------
2022-09-27T15:57:22.3833788Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:57:22.3834490Z 
2022-09-27T15:57:22.3834759Z ----------------------------------------------------------------------
2022-09-27T15:57:22.3835104Z Ran 1 test in 0.002s
2022-09-27T15:57:22.3835273Z 
2022-09-27T15:57:22.3835385Z OK (skipped=1)
2022-09-27T15:57:22.3835561Z 
2022-09-27T15:57:22.3835689Z Generating XML reports...
2022-09-27T15:57:22.3871107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155722.xml
2022-09-27T15:57:22.5364692Z skip: Skipped due to small world size. (4.463s)
2022-09-27T15:57:22.5364956Z 
2022-09-27T15:57:22.5365327Z ----------------------------------------------------------------------
2022-09-27T15:57:22.5365665Z Ran 1 test in 4.463s
2022-09-27T15:57:22.5365837Z 
2022-09-27T15:57:22.5365930Z OK (skipped=1)
2022-09-27T15:57:22.5366092Z 
2022-09-27T15:57:22.5366220Z Generating XML reports...
2022-09-27T15:57:22.5402142Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155718.xml
2022-09-27T15:57:22.7676491Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:22.7698870Z 
2022-09-27T15:57:22.7699340Z Running tests...
2022-09-27T15:57:22.7699865Z ----------------------------------------------------------------------
2022-09-27T15:57:24.3370881Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:24.3371580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:24.3373533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:24.3374031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:24.4478610Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:24.4671603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22988
2022-09-27T15:57:24.4677781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22989
2022-09-27T15:57:24.5954348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:24.5955277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:24.5956600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:24.5957095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:24.8221366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:24.8243402Z 
2022-09-27T15:57:24.8244020Z Running tests...
2022-09-27T15:57:24.8244538Z ----------------------------------------------------------------------
2022-09-27T15:57:24.8262962Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:57:24.8263348Z 
2022-09-27T15:57:24.8263684Z ----------------------------------------------------------------------
2022-09-27T15:57:24.8264029Z Ran 1 test in 0.002s
2022-09-27T15:57:24.8264223Z 
2022-09-27T15:57:24.8264336Z OK (skipped=1)
2022-09-27T15:57:24.8264494Z 
2022-09-27T15:57:24.8264602Z Generating XML reports...
2022-09-27T15:57:24.8298635Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155724.xml
2022-09-27T15:57:25.1197369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:25.1241292Z 
2022-09-27T15:57:25.1242320Z Running tests...
2022-09-27T15:57:25.1243813Z ----------------------------------------------------------------------
2022-09-27T15:57:25.1278765Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.004s)
2022-09-27T15:57:25.1280143Z 
2022-09-27T15:57:25.1281516Z ----------------------------------------------------------------------
2022-09-27T15:57:25.1282221Z Ran 1 test in 0.004s
2022-09-27T15:57:25.1282536Z 
2022-09-27T15:57:25.1282758Z OK (skipped=1)
2022-09-27T15:57:25.1283125Z 
2022-09-27T15:57:25.1283403Z Generating XML reports...
2022-09-27T15:57:25.1348194Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155725.xml
2022-09-27T15:57:26.1191978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:26.1192751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:26.1195229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:26.1195803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:26.1325733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:26.1326216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:26.1329659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:26.1330412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:26.6431846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:26.6797426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:26.6798257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:26.6836860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:26.6844229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:26.6908946Z STAGE:2022-09-27 15:57:26 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:26.7809256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:26.7873084Z STAGE:2022-09-27 15:57:26 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:26.8141563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:26.8142069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:26.8144768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:26.8145274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:27.1893253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:27.1893848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:27.1894794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:27.1895282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:27.3796954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:27.3820651Z 
2022-09-27T15:57:27.3820965Z Running tests...
2022-09-27T15:57:27.3831425Z ----------------------------------------------------------------------
2022-09-27T15:57:27.3841230Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:57:27.3841598Z 
2022-09-27T15:57:27.3842080Z ----------------------------------------------------------------------
2022-09-27T15:57:27.3842428Z Ran 1 test in 0.002s
2022-09-27T15:57:27.3842597Z 
2022-09-27T15:57:27.3842711Z OK (skipped=1)
2022-09-27T15:57:27.3842878Z 
2022-09-27T15:57:27.3843008Z Generating XML reports...
2022-09-27T15:57:27.3878438Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155727.xml
2022-09-27T15:57:27.7171753Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:27.7195725Z 
2022-09-27T15:57:27.7195866Z Running tests...
2022-09-27T15:57:27.7196510Z ----------------------------------------------------------------------
2022-09-27T15:57:27.7219343Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T15:57:27.7219784Z 
2022-09-27T15:57:27.7220181Z ----------------------------------------------------------------------
2022-09-27T15:57:27.7220523Z Ran 1 test in 0.002s
2022-09-27T15:57:27.7220693Z 
2022-09-27T15:57:27.7220787Z OK (skipped=1)
2022-09-27T15:57:27.7220947Z 
2022-09-27T15:57:27.7221074Z Generating XML reports...
2022-09-27T15:57:27.7257085Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155727.xml
2022-09-27T15:57:28.4088986Z [1664294248.408201] [28fd6574525b:22988:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:57:28.4090835Z [1664294248.408201] [28fd6574525b:22988:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:57:28.4233537Z [1664294248.422812] [28fd6574525b:22989:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:57:28.4235529Z [1664294248.422812] [28fd6574525b:22989:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:57:28.8743639Z STAGE:2022-09-27 15:57:28 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:57:28 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:28.8744090Z 
2022-09-27T15:57:28.9964751Z STAGE:2022-09-27 15:57:28 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:57:28 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:28.9965210Z 
2022-09-27T15:57:28.9974535Z STAGE:2022-09-27 15:57:28 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:57:28 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:28.9974941Z 
2022-09-27T15:57:28.9985985Z STAGE:2022-09-27 15:57:28 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:28.9994469Z STAGE:2022-09-27 15:57:28 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0005993Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0007531Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0018784Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0020681Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0031412Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0032657Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0043206Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0048743Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0058852Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0060090Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0072811Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0074181Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0084802Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0086044Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0096182Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0101707Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0111808Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0112993Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0125556Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0126947Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0137031Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0138265Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0148607Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0154261Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0164560Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0165769Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0178467Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0179915Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0189888Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0191137Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0201896Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0207494Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0217375Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0218576Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0230943Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0232345Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:29.0242521Z STAGE:2022-09-27 15:57:29 22989:22989 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.0243724Z STAGE:2022-09-27 15:57:29 22988:22988 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:29.4746879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:29.4747397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:29.4748956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:29.4749448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:29.7253513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:29.7254011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:29.7256387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:29.7256887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:29.9750052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:29.9774040Z 
2022-09-27T15:57:29.9774241Z Running tests...
2022-09-27T15:57:29.9775375Z ----------------------------------------------------------------------
2022-09-27T15:57:29.9793993Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:57:29.9794681Z 
2022-09-27T15:57:29.9795176Z ----------------------------------------------------------------------
2022-09-27T15:57:29.9795810Z Ran 1 test in 0.002s
2022-09-27T15:57:29.9796103Z 
2022-09-27T15:57:29.9796301Z OK (skipped=1)
2022-09-27T15:57:29.9796598Z 
2022-09-27T15:57:29.9796844Z Generating XML reports...
2022-09-27T15:57:29.9831491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155729.xml
2022-09-27T15:57:30.1812896Z ok (7.411s)
2022-09-27T15:57:30.1813087Z 
2022-09-27T15:57:30.1814010Z ----------------------------------------------------------------------
2022-09-27T15:57:30.1814398Z Ran 1 test in 7.411s
2022-09-27T15:57:30.1814576Z 
2022-09-27T15:57:30.1814676Z OK
2022-09-27T15:57:30.1814818Z 
2022-09-27T15:57:30.1814954Z Generating XML reports...
2022-09-27T15:57:30.1851674Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155722.xml
2022-09-27T15:57:30.2229150Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:30.2253851Z 
2022-09-27T15:57:30.2254379Z Running tests...
2022-09-27T15:57:30.2254836Z ----------------------------------------------------------------------
2022-09-27T15:57:30.2273619Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T15:57:30.2273945Z 
2022-09-27T15:57:30.2274851Z ----------------------------------------------------------------------
2022-09-27T15:57:30.2275189Z Ran 1 test in 0.002s
2022-09-27T15:57:30.2275363Z 
2022-09-27T15:57:30.2278024Z OK (skipped=1)
2022-09-27T15:57:30.2278817Z 
2022-09-27T15:57:30.2283328Z Generating XML reports...
2022-09-27T15:57:30.2314745Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155730.xml
2022-09-27T15:57:31.9899437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:31.9900007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:31.9901302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:31.9902116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:32.2344925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:32.2345420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:32.2347901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:32.2348382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:32.3536842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:32.3537342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:32.3539697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:32.3540174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:32.4949884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:32.4973377Z 
2022-09-27T15:57:32.4973658Z Running tests...
2022-09-27T15:57:32.4974100Z ----------------------------------------------------------------------
2022-09-27T15:57:32.4993623Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:57:32.4993986Z 
2022-09-27T15:57:32.4994261Z ----------------------------------------------------------------------
2022-09-27T15:57:32.4994596Z Ran 1 test in 0.002s
2022-09-27T15:57:32.4994765Z 
2022-09-27T15:57:32.4994880Z OK (skipped=1)
2022-09-27T15:57:32.4995042Z 
2022-09-27T15:57:32.4995187Z Generating XML reports...
2022-09-27T15:57:32.5029645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155732.xml
2022-09-27T15:57:32.7241750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:32.7264633Z 
2022-09-27T15:57:32.7265002Z Running tests...
2022-09-27T15:57:32.7265471Z ----------------------------------------------------------------------
2022-09-27T15:57:32.7288878Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-09-27T15:57:32.7289201Z 
2022-09-27T15:57:32.7289469Z ----------------------------------------------------------------------
2022-09-27T15:57:32.7289808Z Ran 1 test in 0.002s
2022-09-27T15:57:32.7289978Z 
2022-09-27T15:57:32.7290091Z OK (skipped=1)
2022-09-27T15:57:32.7290251Z 
2022-09-27T15:57:32.7290380Z Generating XML reports...
2022-09-27T15:57:32.7326861Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155732.xml
2022-09-27T15:57:32.8468129Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:32.8491096Z 
2022-09-27T15:57:32.8491411Z Running tests...
2022-09-27T15:57:32.8492139Z ----------------------------------------------------------------------
2022-09-27T15:57:32.8512739Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and Nccl backend supports CUDA allReduce (0.002s)
2022-09-27T15:57:32.8513109Z 
2022-09-27T15:57:32.8513423Z ----------------------------------------------------------------------
2022-09-27T15:57:32.8513763Z Ran 1 test in 0.002s
2022-09-27T15:57:32.8513933Z 
2022-09-27T15:57:32.8514045Z OK (skipped=1)
2022-09-27T15:57:32.8514209Z 
2022-09-27T15:57:32.8514338Z Generating XML reports...
2022-09-27T15:57:32.8549167Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155732.xml
2022-09-27T15:57:34.4878815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:34.4879696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:34.4881423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:34.4881925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:34.7585166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:34.7585832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:34.7588041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:34.7588810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:34.8371328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:34.8372028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:34.8373545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:34.8374033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:34.9685888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:34.9708257Z 
2022-09-27T15:57:34.9708527Z Running tests...
2022-09-27T15:57:34.9709079Z ----------------------------------------------------------------------
2022-09-27T15:57:34.9727396Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T15:57:34.9727910Z 
2022-09-27T15:57:34.9728296Z ----------------------------------------------------------------------
2022-09-27T15:57:34.9728615Z Ran 1 test in 0.002s
2022-09-27T15:57:34.9728792Z 
2022-09-27T15:57:34.9728905Z OK (skipped=1)
2022-09-27T15:57:34.9729183Z 
2022-09-27T15:57:34.9729425Z Generating XML reports...
2022-09-27T15:57:34.9763046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155734.xml
2022-09-27T15:57:35.2464424Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:35.2487703Z 
2022-09-27T15:57:35.2488060Z Running tests...
2022-09-27T15:57:35.2488687Z ----------------------------------------------------------------------
2022-09-27T15:57:35.2509781Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-09-27T15:57:35.2510744Z 
2022-09-27T15:57:35.2511208Z ----------------------------------------------------------------------
2022-09-27T15:57:35.2511547Z Ran 1 test in 0.002s
2022-09-27T15:57:35.2512011Z 
2022-09-27T15:57:35.2512140Z OK (skipped=1)
2022-09-27T15:57:35.2512297Z 
2022-09-27T15:57:35.2512425Z Generating XML reports...
2022-09-27T15:57:35.2546016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155735.xml
2022-09-27T15:57:35.3449787Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:35.3473606Z 
2022-09-27T15:57:35.3474013Z Running tests...
2022-09-27T15:57:35.3474486Z ----------------------------------------------------------------------
2022-09-27T15:57:36.8692172Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:36.8693171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:36.8694266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:36.8695076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:36.9549599Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:36.9754178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23467
2022-09-27T15:57:36.9759938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23468
2022-09-27T15:57:37.1919590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:37.1920515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:37.1922812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:37.1923296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:37.3520706Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:37.3542848Z 
2022-09-27T15:57:37.3543227Z Running tests...
2022-09-27T15:57:37.3543697Z ----------------------------------------------------------------------
2022-09-27T15:57:37.3563468Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T15:57:37.3563830Z 
2022-09-27T15:57:37.3564467Z ----------------------------------------------------------------------
2022-09-27T15:57:37.3564806Z Ran 1 test in 0.002s
2022-09-27T15:57:37.3564972Z 
2022-09-27T15:57:37.3565082Z OK (skipped=1)
2022-09-27T15:57:37.3565247Z 
2022-09-27T15:57:37.3565375Z Generating XML reports...
2022-09-27T15:57:37.3599170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155737.xml
2022-09-27T15:57:37.7402541Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:37.7424805Z 
2022-09-27T15:57:37.7425394Z Running tests...
2022-09-27T15:57:37.7425895Z ----------------------------------------------------------------------
2022-09-27T15:57:38.5876356Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:38.5877061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:38.5878167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:38.5878695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:38.6355511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:38.6356022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:38.6356953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:38.6357473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:39.0942787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:39.1530675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:39.1531496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:39.1550248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:39.1557455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:39.1560711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:57:39.2543545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:39.2546538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:57:39.2547711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:57:39.2576336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:57:39.2614440Z STAGE:2022-09-27 15:57:39 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:39.2636129Z STAGE:2022-09-27 15:57:39 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:39.3349763Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:39.3545855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23576
2022-09-27T15:57:39.3551915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23577
2022-09-27T15:57:39.3971711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:39.3972469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:39.3973992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:39.3974485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:39.9190713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:39.9214470Z 
2022-09-27T15:57:39.9214771Z Running tests...
2022-09-27T15:57:39.9215218Z ----------------------------------------------------------------------
2022-09-27T15:57:40.9529985Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... [1664294260.952475] [28fd6574525b:23468:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:57:40.9530641Z [1664294260.952475] [28fd6574525b:23468:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:57:40.9531109Z [1664294260.952756] [28fd6574525b:23467:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:57:40.9531607Z [1664294260.952756] [28fd6574525b:23467:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:57:41.0324787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:41.0325405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:41.0326705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:41.0327177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:41.0346340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:41.0346948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:41.0350599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:41.0351069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:41.3337200Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.3337647Z 
2022-09-27T15:57:41.4571709Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4579767Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4581275Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4590134Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4592683Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4610652Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4620462Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4620873Z 
2022-09-27T15:57:41.4631098Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4634597Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4644509Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4645108Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4656482Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4658835Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4668434Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4669027Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4679111Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4682564Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4693076Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4693673Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4704982Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4706503Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4717167Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4717574Z 
2022-09-27T15:57:41.4727841Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4731154Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4741277Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4741905Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4752927Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4754565Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4765416Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4765819Z 
2022-09-27T15:57:41.4775827Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4779105Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4788961Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4789553Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4801300Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4802949Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4813070Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4813667Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4824433Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4827598Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:57:41.4837325Z STAGE:2022-09-27 15:57:41 23467:23467 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.4837924Z STAGE:2022-09-27 15:57:41 23468:23468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:57:41.5574062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:41.5700340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:41.5701144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:41.5778430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:41.5785024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:41.5952078Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:41.6161712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23657
2022-09-27T15:57:41.6166622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23658
2022-09-27T15:57:41.6713562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:41.6944502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:57:41.6945072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:57:41.6945871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:57:41.6946579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:57:41.6947129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T15:57:41.6947892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T15:57:41.6948573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T15:57:41.6949288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T15:57:42.1628701Z ok (4.420s)
2022-09-27T15:57:42.1629070Z 
2022-09-27T15:57:42.1629648Z ----------------------------------------------------------------------
2022-09-27T15:57:42.1630001Z Ran 1 test in 4.420s
2022-09-27T15:57:42.1630171Z 
2022-09-27T15:57:42.1630248Z OK
2022-09-27T15:57:42.1630383Z 
2022-09-27T15:57:42.1630518Z Generating XML reports...
2022-09-27T15:57:42.1665839Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155737.xml
2022-09-27T15:57:42.6883297Z ok (7.341s)
2022-09-27T15:57:42.6883524Z 
2022-09-27T15:57:42.6884181Z ----------------------------------------------------------------------
2022-09-27T15:57:42.6884548Z Ran 1 test in 7.341s
2022-09-27T15:57:42.6884700Z 
2022-09-27T15:57:42.6885164Z OK
2022-09-27T15:57:42.6885306Z 
2022-09-27T15:57:42.6885447Z Generating XML reports...
2022-09-27T15:57:42.6921024Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155735.xml
2022-09-27T15:57:43.2589285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:43.2589876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:43.2591535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:43.2592182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:43.2696080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:43.2696544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:43.2699847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:43.2700347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:43.7929912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:43.7930465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:43.7931259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:43.7931968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:43.8037871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:43.8989706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:44.4738836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:44.4739364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:44.4740531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:44.4741007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:44.8918552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:44.8919067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:44.8920752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:44.8921258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:45.0010681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:45.0035015Z 
2022-09-27T15:57:45.0035303Z Running tests...
2022-09-27T15:57:45.0035763Z ----------------------------------------------------------------------
2022-09-27T15:57:45.4193100Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:45.4216110Z 
2022-09-27T15:57:45.4216389Z Running tests...
2022-09-27T15:57:45.4216836Z ----------------------------------------------------------------------
2022-09-27T15:57:46.2502820Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:57:46.2503489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:57:46.2504276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:57:46.2505234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:57:46.6957063Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:46.7166956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23825
2022-09-27T15:57:46.7172590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23826
2022-09-27T15:57:46.9284067Z ok (7.007s)
2022-09-27T15:57:46.9284295Z 
2022-09-27T15:57:46.9284709Z ----------------------------------------------------------------------
2022-09-27T15:57:46.9285055Z Ran 1 test in 7.007s
2022-09-27T15:57:46.9285219Z 
2022-09-27T15:57:46.9285299Z OK
2022-09-27T15:57:46.9285436Z 
2022-09-27T15:57:46.9285591Z Generating XML reports...
2022-09-27T15:57:46.9321941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155739.xml
2022-09-27T15:57:47.0514944Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:47.0720134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23828
2022-09-27T15:57:47.0725462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23829
2022-09-27T15:57:48.4882883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:48.4883425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:48.4884837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:48.4885393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:48.5053560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:48.5054060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:48.5056884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:48.5057518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:48.8370319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:48.8370839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:48.8372107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:48.8372927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:48.8582456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:48.8583224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:48.8586824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:48.8587677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:49.0159748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:49.0161044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:49.0161865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:49.0162553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:49.0168774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:49.0169825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:49.0825682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:49.0826584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:49.0828140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:49.0829007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:49.3606397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:49.3906601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:49.3907935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:49.3910895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:49.3917767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:49.4921344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:49.6146616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:49.6169799Z 
2022-09-27T15:57:49.6170211Z Running tests...
2022-09-27T15:57:49.6171172Z ----------------------------------------------------------------------
2022-09-27T15:57:49.8798406Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Skipped due to small world size. (4.458s)
2022-09-27T15:57:49.8799173Z 
2022-09-27T15:57:49.8799824Z ----------------------------------------------------------------------
2022-09-27T15:57:49.8800535Z Ran 1 test in 4.458s
2022-09-27T15:57:49.8800716Z 
2022-09-27T15:57:49.8800829Z OK (skipped=1)
2022-09-27T15:57:49.8800988Z 
2022-09-27T15:57:49.8801118Z Generating XML reports...
2022-09-27T15:57:49.8835398Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155745.xml
2022-09-27T15:57:51.0267339Z ok (6.023s)
2022-09-27T15:57:51.0267547Z 
2022-09-27T15:57:51.0267945Z ----------------------------------------------------------------------
2022-09-27T15:57:51.0268298Z Ran 1 test in 6.023s
2022-09-27T15:57:51.0268465Z 
2022-09-27T15:57:51.0268559Z OK
2022-09-27T15:57:51.0268698Z 
2022-09-27T15:57:51.0268817Z Generating XML reports...
2022-09-27T15:57:51.0304419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155744.xml
2022-09-27T15:57:51.3488501Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:51.3693304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24044
2022-09-27T15:57:51.3698311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24045
2022-09-27T15:57:52.1668205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:52.1668747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:52.1671068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:52.1671568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:52.7093229Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:57:52.7116063Z 
2022-09-27T15:57:52.7116292Z Running tests...
2022-09-27T15:57:52.7116991Z ----------------------------------------------------------------------
2022-09-27T15:57:53.0530967Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:53.0531577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:53.0533122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:53.0598659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:53.0599886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:53.0600734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:53.0602679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:53.0603500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:53.2618055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:53.2618632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:53.2619914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:53.2620672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:53.5938068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:53.5948114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:53.5949484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:53.6039329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:53.6058018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:53.7053043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:53.7783119Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:57:53.7806593Z 
2022-09-27T15:57:53.7806908Z Running tests...
2022-09-27T15:57:53.7807876Z ----------------------------------------------------------------------
2022-09-27T15:57:54.0764704Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: Need at least 3 CUDA devices (4.459s)
2022-09-27T15:57:54.0765530Z 
2022-09-27T15:57:54.0766002Z ----------------------------------------------------------------------
2022-09-27T15:57:54.0766343Z Ran 1 test in 4.459s
2022-09-27T15:57:54.0766516Z 
2022-09-27T15:57:54.0766609Z OK (skipped=1)
2022-09-27T15:57:54.0766770Z 
2022-09-27T15:57:54.0766897Z Generating XML reports...
2022-09-27T15:57:54.0802168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155749.xml
2022-09-27T15:57:54.3208941Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:54.3411072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24155
2022-09-27T15:57:54.3416573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24156
2022-09-27T15:57:55.5283206Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:57:55.5482128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24252
2022-09-27T15:57:55.5488608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24253
2022-09-27T15:57:56.0621250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:56.0621799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:56.0623564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:56.0624058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:56.0840943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:56.0841983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:56.0845066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:56.0845561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:56.4158776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:56.4159806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:56.4161367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:56.4162397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:56.5927094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:56.6251270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:56.6252090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:56.6332488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:56.6339497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:56.7266136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:56.9239968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:56.9263734Z 
2022-09-27T15:57:56.9264070Z Running tests...
2022-09-27T15:57:56.9264800Z ----------------------------------------------------------------------
2022-09-27T15:57:56.9284344Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-09-27T15:57:56.9284672Z 
2022-09-27T15:57:56.9284954Z ----------------------------------------------------------------------
2022-09-27T15:57:56.9285296Z Ran 1 test in 0.002s
2022-09-27T15:57:56.9285463Z 
2022-09-27T15:57:56.9285572Z OK (skipped=1)
2022-09-27T15:57:56.9285728Z 
2022-09-27T15:57:56.9285854Z Generating XML reports...
2022-09-27T15:57:56.9321312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155756.xml
2022-09-27T15:57:57.1926153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:57.1926677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:57.1928327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:57.1928816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:57.1938634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:57.1939105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:57.1942739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:57.1943220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:57.7378623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:57:57.7408941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:57:57.7409777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:57.7479967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:57:57.7487566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:57:57.8424160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:57:58.0384286Z [1664294278.037462] [28fd6574525b:24155:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:57:58.0385660Z [1664294278.037462] [28fd6574525b:24155:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:57:58.0436097Z [1664294278.043051] [28fd6574525b:24156:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:57:58.0437387Z [1664294278.043051] [28fd6574525b:24156:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:57:58.2567469Z skip: Skipped due to small world size. (4.476s)
2022-09-27T15:57:58.2567780Z 
2022-09-27T15:57:58.2568167Z ----------------------------------------------------------------------
2022-09-27T15:57:58.2568510Z Ran 1 test in 4.476s
2022-09-27T15:57:58.2568659Z 
2022-09-27T15:57:58.2568768Z OK (skipped=1)
2022-09-27T15:57:58.2568923Z 
2022-09-27T15:57:58.2569049Z Generating XML reports...
2022-09-27T15:57:58.2766748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155753.xml
2022-09-27T15:57:58.9527257Z ok (6.241s)
2022-09-27T15:57:58.9527504Z 
2022-09-27T15:57:58.9528143Z ----------------------------------------------------------------------
2022-09-27T15:57:58.9528531Z Ran 1 test in 6.241s
2022-09-27T15:57:58.9528686Z 
2022-09-27T15:57:58.9528783Z OK
2022-09-27T15:57:58.9528920Z 
2022-09-27T15:57:58.9529076Z Generating XML reports...
2022-09-27T15:57:58.9549080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:57:58.9549561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:57:58.9552991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:57:58.9553496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:57:58.9562860Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155752.xml
2022-09-27T15:57:59.4712163Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:57:59.4736822Z 
2022-09-27T15:57:59.4737303Z Running tests...
2022-09-27T15:57:59.4737807Z ----------------------------------------------------------------------
2022-09-27T15:58:00.5157403Z   test_barrier (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:00.5158031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:00.5159094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:00.5159571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:01.0250309Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:01.0273362Z 
2022-09-27T15:58:01.0273716Z Running tests...
2022-09-27T15:58:01.0274162Z ----------------------------------------------------------------------
2022-09-27T15:58:01.0322280Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:01.0322848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:01.0327145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:01.0327990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:01.0890323Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:01.1095704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24442
2022-09-27T15:58:01.1102100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24443
2022-09-27T15:58:01.5359946Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:01.5383422Z 
2022-09-27T15:58:01.5384013Z Running tests...
2022-09-27T15:58:01.5384508Z ----------------------------------------------------------------------
2022-09-27T15:58:01.5404344Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T15:58:01.5404828Z 
2022-09-27T15:58:01.5405113Z ----------------------------------------------------------------------
2022-09-27T15:58:01.5405461Z Ran 1 test in 0.002s
2022-09-27T15:58:01.5405628Z 
2022-09-27T15:58:01.5405738Z OK (skipped=1)
2022-09-27T15:58:01.5405895Z 
2022-09-27T15:58:01.5406004Z Generating XML reports...
2022-09-27T15:58:01.5439716Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155801.xml
2022-09-27T15:58:02.7037374Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:02.7237633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24539
2022-09-27T15:58:02.7244124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24540
2022-09-27T15:58:02.7989144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:02.7989683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:02.7991853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:02.7992349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:02.8542542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:02.8543053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:02.8571491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:02.8571992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:03.3759720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:03.3762784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:03.3766011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:03.3768083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:03.3815309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:03.3837117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:03.6280826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:03.6281349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:03.6284230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:03.6284755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:04.1162361Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:04.1184974Z 
2022-09-27T15:58:04.1185398Z Running tests...
2022-09-27T15:58:04.1185859Z ----------------------------------------------------------------------
2022-09-27T15:58:04.1204226Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T15:58:04.1204619Z 
2022-09-27T15:58:04.1204887Z ----------------------------------------------------------------------
2022-09-27T15:58:04.1205203Z Ran 1 test in 0.002s
2022-09-27T15:58:04.1205374Z 
2022-09-27T15:58:04.1205485Z OK (skipped=1)
2022-09-27T15:58:04.1205644Z 
2022-09-27T15:58:04.1205770Z Generating XML reports...
2022-09-27T15:58:04.1239173Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155804.xml
2022-09-27T15:58:04.4112591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:04.4113110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:04.4115168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:04.4115644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:04.4287294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:04.4287791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:04.4289049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:04.4289515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:04.6182524Z ok (5.144s)
2022-09-27T15:58:04.6182914Z 
2022-09-27T15:58:04.6183538Z ----------------------------------------------------------------------
2022-09-27T15:58:04.6184163Z Ran 1 test in 5.145s
2022-09-27T15:58:04.6184487Z 
2022-09-27T15:58:04.6184632Z OK
2022-09-27T15:58:04.6184888Z 
2022-09-27T15:58:04.6185133Z Generating XML reports...
2022-09-27T15:58:04.6221344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155759.xml
2022-09-27T15:58:04.9418291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:04.9481001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:04.9481801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:04.9519414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:04.9526893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:05.0580597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:06.1055869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:06.1056632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:06.1057491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:06.1057986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:06.6060193Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:06.6083063Z 
2022-09-27T15:58:06.6083344Z Running tests...
2022-09-27T15:58:06.6084038Z ----------------------------------------------------------------------
2022-09-27T15:58:06.6101960Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T15:58:06.6103152Z 
2022-09-27T15:58:06.6103454Z ----------------------------------------------------------------------
2022-09-27T15:58:06.6103773Z Ran 1 test in 0.002s
2022-09-27T15:58:06.6103942Z 
2022-09-27T15:58:06.6104054Z OK (skipped=1)
2022-09-27T15:58:06.6104211Z 
2022-09-27T15:58:06.6104337Z Generating XML reports...
2022-09-27T15:58:06.6138109Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155806.xml
2022-09-27T15:58:06.7803449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:06.7803963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:06.7806182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:06.7806671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:06.8341347Z ok (5.807s)
2022-09-27T15:58:06.8341554Z 
2022-09-27T15:58:06.8342433Z ----------------------------------------------------------------------
2022-09-27T15:58:06.8342827Z Ran 1 test in 5.807s
2022-09-27T15:58:06.8342984Z 
2022-09-27T15:58:06.8343081Z OK
2022-09-27T15:58:06.8343231Z 
2022-09-27T15:58:06.8343367Z Generating XML reports...
2022-09-27T15:58:06.8379833Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155801.xml
2022-09-27T15:58:07.2880663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:07.2904190Z 
2022-09-27T15:58:07.2904648Z Running tests...
2022-09-27T15:58:07.2905133Z ----------------------------------------------------------------------
2022-09-27T15:58:08.5917339Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:08.5917953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:08.5920445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:08.5921165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:08.9109884Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:08.9307079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24756
2022-09-27T15:58:08.9313257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24757
2022-09-27T15:58:08.9359610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:08.9360468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:08.9363772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:08.9364286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:09.0977679Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:09.1001611Z 
2022-09-27T15:58:09.1001889Z Running tests...
2022-09-27T15:58:09.1002349Z ----------------------------------------------------------------------
2022-09-27T15:58:09.4289006Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:09.4311180Z 
2022-09-27T15:58:09.4311619Z Running tests...
2022-09-27T15:58:09.4312104Z ----------------------------------------------------------------------
2022-09-27T15:58:10.5779501Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:10.5780145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:10.5781720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:10.5782222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:10.5832597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:10.5833063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:10.5837473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:10.5837967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:10.7648759Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:10.7849049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24826
2022-09-27T15:58:10.7854748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24827
2022-09-27T15:58:11.0389299Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:11.0581882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24829
2022-09-27T15:58:11.0587518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24830
2022-09-27T15:58:11.1298942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:11.1299545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:11.1300359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:11.1301096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:11.1407766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:11.2313526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:12.4858862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:12.4859382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:12.4861463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:12.4861966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:12.5324363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:12.5324978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:12.5327599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:12.5328118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:12.7966512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:12.7967008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:12.7967598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:12.7968068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:12.7968875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:12.7969365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:12.7969970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:12.7970444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:12.9966144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:13.0337610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:13.0338423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:13.0371304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:13.0377790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:13.1352472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:13.3151866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:13.3256410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:13.3257240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:13.3356142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:13.3363995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:13.4270347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:13.9421256Z ok (6.651s)
2022-09-27T15:58:13.9421486Z 
2022-09-27T15:58:13.9421892Z ----------------------------------------------------------------------
2022-09-27T15:58:13.9422246Z Ran 1 test in 6.652s
2022-09-27T15:58:13.9422416Z 
2022-09-27T15:58:13.9422525Z OK
2022-09-27T15:58:13.9422645Z 
2022-09-27T15:58:13.9422779Z Generating XML reports...
2022-09-27T15:58:13.9458685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155807.xml
2022-09-27T15:58:13.9658526Z ok (4.534s)
2022-09-27T15:58:13.9658720Z 
2022-09-27T15:58:13.9659083Z ----------------------------------------------------------------------
2022-09-27T15:58:13.9659415Z Ran 1 test in 4.535s
2022-09-27T15:58:13.9659590Z 
2022-09-27T15:58:13.9659685Z OK
2022-09-27T15:58:13.9659827Z 
2022-09-27T15:58:13.9659962Z Generating XML reports...
2022-09-27T15:58:13.9695676Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155809.xml
2022-09-27T15:58:14.4097223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6srdodz
2022-09-27T15:58:14.4098722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6srdodz/_remote_module_non_scriptable.py
2022-09-27T15:58:14.5197229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3efr1pd_
2022-09-27T15:58:14.5199257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3efr1pd_/_remote_module_non_scriptable.py
2022-09-27T15:58:14.5464649Z [1664294294.546000] [28fd6574525b:24827:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:14.5465178Z [1664294294.546000] [28fd6574525b:24827:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:14.5465643Z [1664294294.546039] [28fd6574525b:24826:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:14.5466120Z [1664294294.546039] [28fd6574525b:24826:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:15.3563232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:15.3577294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:15.8969386Z ok (6.796s)
2022-09-27T15:58:15.8969968Z 
2022-09-27T15:58:15.8970359Z ----------------------------------------------------------------------
2022-09-27T15:58:15.8970708Z Ran 1 test in 6.797s
2022-09-27T15:58:15.8970857Z 
2022-09-27T15:58:15.8970954Z OK
2022-09-27T15:58:15.8971092Z 
2022-09-27T15:58:15.8971231Z Generating XML reports...
2022-09-27T15:58:15.9005755Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155809.xml
2022-09-27T15:58:16.1663381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:16.1663867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:16.1665713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:16.1666206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:16.1701656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:16.1702120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:16.1706129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:16.1706617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:16.6599107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:16.6621675Z 
2022-09-27T15:58:16.6621965Z Running tests...
2022-09-27T15:58:16.6622397Z ----------------------------------------------------------------------
2022-09-27T15:58:16.6895967Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:16.6919869Z 
2022-09-27T15:58:16.6920353Z Running tests...
2022-09-27T15:58:16.6921155Z ----------------------------------------------------------------------
2022-09-27T15:58:17.9047709Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:17.9048309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:17.9049197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:17.9049686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:18.2612481Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:18.2816296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25094
2022-09-27T15:58:18.2821722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25095
2022-09-27T15:58:18.3607040Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:18.3808170Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25097
2022-09-27T15:58:18.3813896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25098
2022-09-27T15:58:18.4313123Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:18.4335524Z 
2022-09-27T15:58:18.4335882Z Running tests...
2022-09-27T15:58:18.4336302Z ----------------------------------------------------------------------
2022-09-27T15:58:20.0136764Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:20.0137496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:20.0138643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:20.0139158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:20.0151342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:20.0151815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:20.0155287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:20.0155763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:20.0540278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:20.0540789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:20.0543652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:20.0544146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:20.1149038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:20.1149555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:20.1151878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:20.1152366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:20.1297874Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:20.1500889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25229
2022-09-27T15:58:20.1506087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25230
2022-09-27T15:58:20.5623929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:20.5624492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:20.5625273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:20.5625997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:20.5632875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:20.5633590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:20.5634099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:20.5740209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:58:20.5741074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:58:20.5742093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:58:20.5743120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:58:20.6255882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:20.6256637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:20.6345798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:20.6352811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:20.7269391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:21.8401614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:21.8402991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:21.8404201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:21.8405136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:21.8498589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:21.8499510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:21.8503176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:21.8504145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:21.8912245Z ok (5.229s)
2022-09-27T15:58:21.8912637Z 
2022-09-27T15:58:21.8913341Z ----------------------------------------------------------------------
2022-09-27T15:58:21.8913996Z Ran 1 test in 5.229s
2022-09-27T15:58:21.8914335Z 
2022-09-27T15:58:21.8914510Z OK
2022-09-27T15:58:21.8914774Z 
2022-09-27T15:58:21.8915000Z Generating XML reports...
2022-09-27T15:58:21.8972137Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155816.xml
2022-09-27T15:58:22.4178536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:22.4179096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:22.4179908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:22.4180622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:22.4187631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:22.4188128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:22.7918285Z ok (6.099s)
2022-09-27T15:58:22.7918661Z 
2022-09-27T15:58:22.7919334Z ----------------------------------------------------------------------
2022-09-27T15:58:22.7919964Z Ran 1 test in 6.100s
2022-09-27T15:58:22.7920581Z 
2022-09-27T15:58:22.7920766Z OK
2022-09-27T15:58:22.7921042Z 
2022-09-27T15:58:22.7921287Z Generating XML reports...
2022-09-27T15:58:22.8017082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155816.xml
2022-09-27T15:58:23.8486186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu5cmy480
2022-09-27T15:58:23.8487746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu5cmy480/_remote_module_non_scriptable.py
2022-09-27T15:58:23.8707799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4n6mf82s
2022-09-27T15:58:23.8710196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4n6mf82s/_remote_module_non_scriptable.py
2022-09-27T15:58:23.9058859Z [1664294303.905504] [28fd6574525b:25229:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:23.9059379Z [1664294303.905504] [28fd6574525b:25229:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:23.9066085Z [1664294303.906340] [28fd6574525b:25230:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:23.9066584Z [1664294303.906340] [28fd6574525b:25230:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:24.1759405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:24.1759920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:24.1761549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:24.1762265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:24.6922025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:24.6945196Z 
2022-09-27T15:58:24.6945520Z Running tests...
2022-09-27T15:58:24.6945974Z ----------------------------------------------------------------------
2022-09-27T15:58:24.7268989Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:24.7269607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:24.7276290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:24.7276799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:24.9481886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:24.9482411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:24.9483738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:24.9484209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:25.2618676Z ok (6.828s)
2022-09-27T15:58:25.2618982Z 
2022-09-27T15:58:25.2619571Z ----------------------------------------------------------------------
2022-09-27T15:58:25.2620057Z Ran 1 test in 6.828s
2022-09-27T15:58:25.2620225Z 
2022-09-27T15:58:25.2620319Z OK
2022-09-27T15:58:25.2620455Z 
2022-09-27T15:58:25.2620599Z Generating XML reports...
2022-09-27T15:58:25.2655863Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155818.xml
2022-09-27T15:58:25.4652259Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:25.4676268Z 
2022-09-27T15:58:25.4676934Z Running tests...
2022-09-27T15:58:25.4677482Z ----------------------------------------------------------------------
2022-09-27T15:58:26.4665665Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:26.4869963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25444
2022-09-27T15:58:26.4875227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25445
2022-09-27T15:58:27.1316198Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:27.1865927Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25509
2022-09-27T15:58:27.1878321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25510
2022-09-27T15:58:27.3667381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:27.3667924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:27.3670179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:27.3670671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:27.8716944Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:27.8756887Z 
2022-09-27T15:58:27.8761693Z Running tests...
2022-09-27T15:58:27.8766824Z ----------------------------------------------------------------------
2022-09-27T15:58:28.1927192Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:28.1927805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:28.1930653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:28.1931161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:28.2080364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:28.2080859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:28.2084251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:28.2084753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:28.7505823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:28.7506359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:28.7507188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:28.7507911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:28.7614968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:28.8520757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:28.8976461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:28.8977188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:28.8978477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:28.8978949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:28.9305816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:28.9306386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:28.9308059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:28.9308527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:29.1936530Z skip: Skipped due to small world size. (4.499s)
2022-09-27T15:58:29.1936797Z 
2022-09-27T15:58:29.1937178Z ----------------------------------------------------------------------
2022-09-27T15:58:29.1937523Z Ran 1 test in 4.499s
2022-09-27T15:58:29.1937686Z 
2022-09-27T15:58:29.1937778Z OK (skipped=1)
2022-09-27T15:58:29.1937937Z 
2022-09-27T15:58:29.1938328Z Generating XML reports...
2022-09-27T15:58:29.1975096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155824.xml
2022-09-27T15:58:29.4261195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:29.4690395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:29.4691231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:29.4767985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:29.4774893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:29.4864640Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:29.5064795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25593
2022-09-27T15:58:29.5070734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25594
2022-09-27T15:58:29.5703953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:31.2003313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:31.2003820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:31.2005036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:31.2005523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:31.2606548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:31.2607058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:31.2609396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:31.2609875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:31.4291131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:31.4291648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:31.4292427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:31.4292897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:31.6977903Z ok (6.230s)
2022-09-27T15:58:31.6978132Z 
2022-09-27T15:58:31.6978515Z ----------------------------------------------------------------------
2022-09-27T15:58:31.6978860Z Ran 1 test in 6.230s
2022-09-27T15:58:31.6979025Z 
2022-09-27T15:58:31.6979101Z OK
2022-09-27T15:58:31.6979236Z 
2022-09-27T15:58:31.6979389Z Generating XML reports...
2022-09-27T15:58:31.7014942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155825.xml
2022-09-27T15:58:31.7272078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:31.7831175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:31.7832011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:31.7881149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:31.7888881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:31.8844719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:31.9404403Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:31.9427915Z 
2022-09-27T15:58:31.9428413Z Running tests...
2022-09-27T15:58:31.9429074Z ----------------------------------------------------------------------
2022-09-27T15:58:33.1285822Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucnorw41
2022-09-27T15:58:33.1287050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucnorw41/_remote_module_non_scriptable.py
2022-09-27T15:58:33.2565369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8r1b0ao
2022-09-27T15:58:33.2566921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8r1b0ao/_remote_module_non_scriptable.py
2022-09-27T15:58:33.2843132Z [1664294313.283803] [28fd6574525b:25593:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:33.2844217Z [1664294313.283803] [28fd6574525b:25593:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:33.2855242Z [1664294313.285180] [28fd6574525b:25594:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:33.2856668Z [1664294313.285180] [28fd6574525b:25594:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:33.5598504Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:33.6128839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25745
2022-09-27T15:58:33.6133970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25746
2022-09-27T15:58:33.7668645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:33.7669584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:33.7671281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:33.7672261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:34.1043214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:34.1044234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:34.1053377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:34.1054364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:34.1177336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:34.1178298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:34.1186815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:34.1187799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:34.2983952Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:34.3023370Z 
2022-09-27T15:58:34.3024394Z Running tests...
2022-09-27T15:58:34.3025715Z ----------------------------------------------------------------------
2022-09-27T15:58:34.8203479Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... ok (6.944s)
2022-09-27T15:58:34.8203849Z 
2022-09-27T15:58:34.8204244Z ----------------------------------------------------------------------
2022-09-27T15:58:34.8204585Z Ran 1 test in 6.945s
2022-09-27T15:58:34.8204754Z 
2022-09-27T15:58:34.8204851Z OK
2022-09-27T15:58:34.8204985Z 
2022-09-27T15:58:34.8205096Z Generating XML reports...
2022-09-27T15:58:34.8239825Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155827.xml
2022-09-27T15:58:35.3165256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:35.3165794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:35.3167728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:35.3168205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:35.3381486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:35.3381966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:35.3385345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:35.3385822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:35.8738621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:35.8739178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:35.8740177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:35.8740861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:35.8747199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:35.8747698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:36.0263281Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:36.0643266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25862
2022-09-27T15:58:36.0648289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25863
2022-09-27T15:58:36.3200177Z skip: Skipped due to small world size. (4.377s)
2022-09-27T15:58:36.3200668Z 
2022-09-27T15:58:36.3201069Z ----------------------------------------------------------------------
2022-09-27T15:58:36.3201420Z Ran 1 test in 4.377s
2022-09-27T15:58:36.3201588Z 
2022-09-27T15:58:36.3201701Z OK (skipped=1)
2022-09-27T15:58:36.3201855Z 
2022-09-27T15:58:36.3201984Z Generating XML reports...
2022-09-27T15:58:36.3237580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155831.xml
2022-09-27T15:58:37.0094809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:37.0095333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:37.0097862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:37.0098389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:37.5576233Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:37.5598238Z 
2022-09-27T15:58:37.5598504Z Running tests...
2022-09-27T15:58:37.5598926Z ----------------------------------------------------------------------
2022-09-27T15:58:37.6709205Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:37.6709812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:37.6711064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:37.6711556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:37.7850691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:37.7851349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:37.7852270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:37.7852771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:38.1833217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:38.3041211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:38.3042064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:38.3049023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:38.3056174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:38.4054265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:38.4378597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:38.4379767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:38.4380747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:38.4381217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:38.8721302Z ok (4.569s)
2022-09-27T15:58:38.8721520Z 
2022-09-27T15:58:38.8721899Z ----------------------------------------------------------------------
2022-09-27T15:58:38.8722237Z Ran 1 test in 4.570s
2022-09-27T15:58:38.8722407Z 
2022-09-27T15:58:38.8722483Z OK
2022-09-27T15:58:38.8722633Z 
2022-09-27T15:58:38.8722768Z Generating XML reports...
2022-09-27T15:58:38.8758294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155834.xml
2022-09-27T15:58:38.9543503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:38.9565668Z 
2022-09-27T15:58:38.9566050Z Running tests...
2022-09-27T15:58:38.9566507Z ----------------------------------------------------------------------
2022-09-27T15:58:39.1213617Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:39.1416458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25971
2022-09-27T15:58:39.1422032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25972
2022-09-27T15:58:40.6134138Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:40.6326300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26068
2022-09-27T15:58:40.6332015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26069
2022-09-27T15:58:40.8219932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:40.8220704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:40.8222376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:40.8222877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:40.8401526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:40.8402171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:40.8405247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:40.8405999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:41.0098251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:41.0098886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:41.0100346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:41.0100824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:41.4005945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:41.4128851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:41.4130377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:41.4207812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:41.4214757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:41.5144811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:41.5527288Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:41.5550858Z 
2022-09-27T15:58:41.5551318Z Running tests...
2022-09-27T15:58:41.5551847Z ----------------------------------------------------------------------
2022-09-27T15:58:41.5570054Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:58:41.5570381Z 
2022-09-27T15:58:41.5570674Z ----------------------------------------------------------------------
2022-09-27T15:58:41.5571009Z Ran 1 test in 0.002s
2022-09-27T15:58:41.5571175Z 
2022-09-27T15:58:41.5571291Z OK (skipped=1)
2022-09-27T15:58:41.5571448Z 
2022-09-27T15:58:41.5571575Z Generating XML reports...
2022-09-27T15:58:41.5606548Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155841.xml
2022-09-27T15:58:42.2577443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:42.2578285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:42.2579109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:42.2579596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:42.2628648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:42.2629121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:42.2632800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:42.2633292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:42.8048819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakddpt6_
2022-09-27T15:58:42.8049560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakddpt6_/_remote_module_non_scriptable.py
2022-09-27T15:58:42.8751876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:42.8752407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:42.8753214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:42.8753923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:42.8761009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:42.8761568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:42.8815976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqmvynlj
2022-09-27T15:58:42.8818302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqmvynlj/_remote_module_non_scriptable.py
2022-09-27T15:58:42.9093408Z [1664294322.908868] [28fd6574525b:25971:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:42.9093925Z [1664294322.908868] [28fd6574525b:25971:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:42.9367624Z [1664294322.936242] [28fd6574525b:25972:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:42.9368196Z [1664294322.936242] [28fd6574525b:25972:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:43.2397581Z skip: Skipped due to small world size. (4.283s)
2022-09-27T15:58:43.2398057Z 
2022-09-27T15:58:43.2398746Z ----------------------------------------------------------------------
2022-09-27T15:58:43.2399825Z Ran 1 test in 4.283s
2022-09-27T15:58:43.2400121Z 
2022-09-27T15:58:43.2400865Z OK (skipped=1)
2022-09-27T15:58:43.2401193Z 
2022-09-27T15:58:43.2401440Z Generating XML reports...
2022-09-27T15:58:43.2435000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155838.xml
2022-09-27T15:58:43.6161935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:43.6162917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:43.6164115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:43.6165087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:43.7292743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:43.7293762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:43.7302891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:43.7303854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:43.7419812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:43.7420775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:43.7430110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:43.7431106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:58:44.1276029Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:44.1298457Z 
2022-09-27T15:58:44.1298964Z Running tests...
2022-09-27T15:58:44.1299487Z ----------------------------------------------------------------------
2022-09-27T15:58:44.3539207Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... ok (6.794s)
2022-09-27T15:58:44.3539791Z 
2022-09-27T15:58:44.3540456Z ----------------------------------------------------------------------
2022-09-27T15:58:44.3541052Z Ran 1 test in 6.794s
2022-09-27T15:58:44.3541372Z 
2022-09-27T15:58:44.3541562Z OK
2022-09-27T15:58:44.3541825Z 
2022-09-27T15:58:44.3542072Z Generating XML reports...
2022-09-27T15:58:44.3575544Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155837.xml
2022-09-27T15:58:45.4722614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:45.4724043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:45.4725321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:45.4726301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:45.8547477Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:45.8713747Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.741s)
2022-09-27T15:58:45.8714897Z 
2022-09-27T15:58:45.8715460Z ----------------------------------------------------------------------
2022-09-27T15:58:45.8716098Z Ran 1 test in 1.741s
2022-09-27T15:58:45.8716409Z 
2022-09-27T15:58:45.8716619Z OK (skipped=1)
2022-09-27T15:58:45.8716923Z 
2022-09-27T15:58:45.8717177Z Generating XML reports...
2022-09-27T15:58:45.8751170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155844.xml
2022-09-27T15:58:46.0306098Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:46.0329853Z 
2022-09-27T15:58:46.0330358Z Running tests...
2022-09-27T15:58:46.0330853Z ----------------------------------------------------------------------
2022-09-27T15:58:46.4895495Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:46.4896140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:46.4897249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:46.4897741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:47.0216653Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:47.0258297Z 
2022-09-27T15:58:47.0260576Z Running tests...
2022-09-27T15:58:47.0261567Z ----------------------------------------------------------------------
2022-09-27T15:58:47.0321683Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.006s)
2022-09-27T15:58:47.0323794Z 
2022-09-27T15:58:47.0328415Z ----------------------------------------------------------------------
2022-09-27T15:58:47.0332053Z Ran 1 test in 0.007s
2022-09-27T15:58:47.0334109Z 
2022-09-27T15:58:47.0337777Z OK (skipped=1)
2022-09-27T15:58:47.0340290Z 
2022-09-27T15:58:47.0344871Z Generating XML reports...
2022-09-27T15:58:47.0410015Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155847.xml
2022-09-27T15:58:47.7124222Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:47.7323219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26297
2022-09-27T15:58:47.7329718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26298
2022-09-27T15:58:47.9459319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:47.9460051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:47.9461485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:47.9461988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:48.5462548Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:48.5505815Z 
2022-09-27T15:58:48.5506271Z Running tests...
2022-09-27T15:58:48.5507646Z ----------------------------------------------------------------------
2022-09-27T15:58:48.5537019Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-09-27T15:58:48.5538009Z 
2022-09-27T15:58:48.5538332Z ----------------------------------------------------------------------
2022-09-27T15:58:48.5538693Z Ran 1 test in 0.003s
2022-09-27T15:58:48.5538866Z 
2022-09-27T15:58:48.5538976Z OK (skipped=1)
2022-09-27T15:58:48.5539134Z 
2022-09-27T15:58:48.5539260Z Generating XML reports...
2022-09-27T15:58:48.5572978Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155848.xml
2022-09-27T15:58:49.1113476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:49.1113988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:49.1115354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:49.1115831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:49.4496430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:49.4497405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:49.4498674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:49.4499174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:49.4631561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:49.4632505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:49.4636763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:49.4637763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:49.6101203Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:49.6125694Z 
2022-09-27T15:58:49.6126572Z Running tests...
2022-09-27T15:58:49.6127501Z ----------------------------------------------------------------------
2022-09-27T15:58:50.0531206Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:50.0532541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:50.0534007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:50.0535458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:50.0538833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:50.0539791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:50.0647390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:58:50.0648561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:58:50.0650084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:58:50.0651635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:58:50.4445150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:50.4446504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:50.4447703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:50.4448656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:50.9405304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:50.9429036Z 
2022-09-27T15:58:50.9429614Z Running tests...
2022-09-27T15:58:50.9430461Z ----------------------------------------------------------------------
2022-09-27T15:58:50.9449850Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T15:58:50.9450479Z 
2022-09-27T15:58:50.9451034Z ----------------------------------------------------------------------
2022-09-27T15:58:50.9451642Z Ran 1 test in 0.002s
2022-09-27T15:58:50.9451956Z 
2022-09-27T15:58:50.9452142Z OK (skipped=1)
2022-09-27T15:58:50.9452448Z 
2022-09-27T15:58:50.9452715Z Generating XML reports...
2022-09-27T15:58:50.9488610Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155850.xml
2022-09-27T15:58:51.1580932Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:51.1789844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26445
2022-09-27T15:58:51.1794569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26446
2022-09-27T15:58:51.4416277Z ok (5.408s)
2022-09-27T15:58:51.4416487Z 
2022-09-27T15:58:51.4416889Z ----------------------------------------------------------------------
2022-09-27T15:58:51.4417213Z Ran 1 test in 5.409s
2022-09-27T15:58:51.4417384Z 
2022-09-27T15:58:51.4417483Z OK
2022-09-27T15:58:51.4417626Z 
2022-09-27T15:58:51.4417762Z Generating XML reports...
2022-09-27T15:58:51.4454558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155846.xml
2022-09-27T15:58:52.9021253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.9021818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.9022630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.9023122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:52.9038582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.9039062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.9043554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.9044046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:52.9276077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:52.9276555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:52.9279998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:52.9280747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:53.4210602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:53.4233351Z 
2022-09-27T15:58:53.4233573Z Running tests...
2022-09-27T15:58:53.4234011Z ----------------------------------------------------------------------
2022-09-27T15:58:53.4253230Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-09-27T15:58:53.4253552Z 
2022-09-27T15:58:53.4253815Z ----------------------------------------------------------------------
2022-09-27T15:58:53.4254422Z Ran 1 test in 0.002s
2022-09-27T15:58:53.4254605Z 
2022-09-27T15:58:53.4254716Z OK (skipped=1)
2022-09-27T15:58:53.4254873Z 
2022-09-27T15:58:53.4254982Z Generating XML reports...
2022-09-27T15:58:53.4288955Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155853.xml
2022-09-27T15:58:53.4332794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:58:53.4385763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:58:53.4387166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:53.4436081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:58:53.4443058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:58:53.5402215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:58:53.6266586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:53.6267441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:53.6268556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:53.6269083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:54.1399229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:54.1421846Z 
2022-09-27T15:58:54.1422088Z Running tests...
2022-09-27T15:58:54.1422545Z ----------------------------------------------------------------------
2022-09-27T15:58:54.1444979Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-09-27T15:58:54.1445428Z 
2022-09-27T15:58:54.1445723Z ----------------------------------------------------------------------
2022-09-27T15:58:54.1446046Z Ran 1 test in 0.002s
2022-09-27T15:58:54.1446215Z 
2022-09-27T15:58:54.1446318Z OK (skipped=1)
2022-09-27T15:58:54.1446475Z 
2022-09-27T15:58:54.1446599Z Generating XML reports...
2022-09-27T15:58:54.1480949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155854.xml
2022-09-27T15:58:54.7754480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspml22wn
2022-09-27T15:58:54.7755291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspml22wn/_remote_module_non_scriptable.py
2022-09-27T15:58:54.8651832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptc1jpeyw
2022-09-27T15:58:54.8652893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptc1jpeyw/_remote_module_non_scriptable.py
2022-09-27T15:58:54.8900550Z [1664294334.889632] [28fd6574525b:26445:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:54.8901088Z [1664294334.889632] [28fd6574525b:26445:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:54.8915914Z [1664294334.891297] [28fd6574525b:26446:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:58:54.8916713Z [1664294334.891297] [28fd6574525b:26446:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:58:55.4546975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:55.4547491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:55.4549487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:55.4550262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:55.7895930Z ok (6.177s)
2022-09-27T15:58:55.7896271Z 
2022-09-27T15:58:55.7896903Z ----------------------------------------------------------------------
2022-09-27T15:58:55.7897481Z Ran 1 test in 6.177s
2022-09-27T15:58:55.7897801Z 
2022-09-27T15:58:55.7897973Z OK
2022-09-27T15:58:55.7898242Z 
2022-09-27T15:58:55.7898481Z Generating XML reports...
2022-09-27T15:58:55.7935556Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155849.xml
2022-09-27T15:58:55.9743591Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:58:55.9766948Z 
2022-09-27T15:58:55.9767454Z Running tests...
2022-09-27T15:58:55.9767970Z ----------------------------------------------------------------------
2022-09-27T15:58:56.2001492Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:56.2002421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:56.2005216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:56.2005693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:56.7075750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:58:56.7098417Z 
2022-09-27T15:58:56.7098866Z Running tests...
2022-09-27T15:58:56.7099293Z ----------------------------------------------------------------------
2022-09-27T15:58:57.6570084Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:57.6743250Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.697s)
2022-09-27T15:58:57.6743866Z 
2022-09-27T15:58:57.6744150Z ----------------------------------------------------------------------
2022-09-27T15:58:57.6744487Z Ran 1 test in 1.697s
2022-09-27T15:58:57.6744653Z 
2022-09-27T15:58:57.6744759Z OK (skipped=1)
2022-09-27T15:58:57.6744918Z 
2022-09-27T15:58:57.6745027Z Generating XML reports...
2022-09-27T15:58:57.6779466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155855.xml
2022-09-27T15:58:57.8685980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:57.8686723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:57.8688241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:57.8688734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:58:58.3540248Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:58:58.3738228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26693
2022-09-27T15:58:58.3744488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26694
2022-09-27T15:58:58.3972739Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:58:58.3995623Z 
2022-09-27T15:58:58.3995842Z Running tests...
2022-09-27T15:58:58.3996412Z ----------------------------------------------------------------------
2022-09-27T15:58:59.9743689Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:58:59.9744356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:58:59.9746187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:58:59.9746686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:00.0275567Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:00.0474041Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26791
2022-09-27T15:59:00.0479051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26792
2022-09-27T15:59:00.1327872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:00.1328643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:00.1329994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:00.1330479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:00.1620587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:00.1621588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:00.1622489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:00.1622968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:00.4859816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:59:00.4883790Z 
2022-09-27T15:59:00.4884077Z Running tests...
2022-09-27T15:59:00.4884527Z ----------------------------------------------------------------------
2022-09-27T15:59:00.6875260Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:00.6899190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:00.6901358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:00.6903941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:00.6981692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:00.7897037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:01.1820442Z skip: Skipped due to small world size. (4.472s)
2022-09-27T15:59:01.1820706Z 
2022-09-27T15:59:01.1821120Z ----------------------------------------------------------------------
2022-09-27T15:59:01.1821485Z Ran 1 test in 4.472s
2022-09-27T15:59:01.1821655Z 
2022-09-27T15:59:01.1821765Z OK (skipped=1)
2022-09-27T15:59:01.1821906Z 
2022-09-27T15:59:01.1822038Z Generating XML reports...
2022-09-27T15:59:01.1882270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155856.xml
2022-09-27T15:59:01.6976721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:01.6977215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:01.6979283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:01.6979780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:01.7358043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:01.7358808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:01.7360429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:01.7360945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:02.1922294Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:02.2091974Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.721s)
2022-09-27T15:59:02.2092608Z 
2022-09-27T15:59:02.2092872Z ----------------------------------------------------------------------
2022-09-27T15:59:02.2093210Z Ran 1 test in 1.721s
2022-09-27T15:59:02.2093378Z 
2022-09-27T15:59:02.2093488Z OK (skipped=1)
2022-09-27T15:59:02.2093646Z 
2022-09-27T15:59:02.2093790Z Generating XML reports...
2022-09-27T15:59:02.2127067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155900.xml
2022-09-27T15:59:02.2459532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:02.2474351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:02.2475474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:02.2561116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:02.2568644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:02.3513006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:03.3886130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:03.3887045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:03.3889409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:03.3890414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:03.6604786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vxwey2z
2022-09-27T15:59:03.6606216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vxwey2z/_remote_module_non_scriptable.py
2022-09-27T15:59:03.7862711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_w3qt0k
2022-09-27T15:59:03.7864231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_w3qt0k/_remote_module_non_scriptable.py
2022-09-27T15:59:03.8112786Z [1664294343.810897] [28fd6574525b:26791:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:03.8113290Z [1664294343.810897] [28fd6574525b:26791:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:03.8176280Z [1664294343.817302] [28fd6574525b:26792:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:03.8176769Z [1664294343.817302] [28fd6574525b:26792:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:03.8960415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:03.8983412Z 
2022-09-27T15:59:03.8983753Z Running tests...
2022-09-27T15:59:03.8984208Z ----------------------------------------------------------------------
2022-09-27T15:59:04.3975864Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:04.3976483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:04.3977398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:04.3977918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:04.5881811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:04.5882348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:04.9281642Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:59:04.9305606Z 
2022-09-27T15:59:04.9306214Z Running tests...
2022-09-27T15:59:04.9307355Z ----------------------------------------------------------------------
2022-09-27T15:59:05.1608227Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... ok (6.761s)
2022-09-27T15:59:05.1608585Z 
2022-09-27T15:59:05.1608981Z ----------------------------------------------------------------------
2022-09-27T15:59:05.1609633Z Ran 1 test in 6.761s
2022-09-27T15:59:05.1609800Z 
2022-09-27T15:59:05.1609897Z OK
2022-09-27T15:59:05.1610032Z 
2022-09-27T15:59:05.1610143Z Generating XML reports...
2022-09-27T15:59:05.1670671Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155858.xml
2022-09-27T15:59:05.5390714Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:05.5589714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26954
2022-09-27T15:59:05.5595612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26955
2022-09-27T15:59:06.7594138Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:06.7758307Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.845s)
2022-09-27T15:59:06.7758911Z 
2022-09-27T15:59:06.7759189Z ----------------------------------------------------------------------
2022-09-27T15:59:06.7759512Z Ran 1 test in 1.845s
2022-09-27T15:59:06.7759678Z 
2022-09-27T15:59:06.7759789Z OK (skipped=1)
2022-09-27T15:59:06.7759946Z 
2022-09-27T15:59:06.7760072Z Generating XML reports...
2022-09-27T15:59:06.7792811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155904.xml
2022-09-27T15:59:07.3045975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:07.3046466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:07.3047855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:07.3048347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:07.3302903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:07.3303401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:07.3305630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:07.3306119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:07.4550574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:07.4551046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:07.4553108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:07.4553876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:07.8556385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:07.8605380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:07.8608178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:07.8656980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:07.8708606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:07.9621767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:08.0066473Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:08.0103805Z 
2022-09-27T15:59:08.0104463Z Running tests...
2022-09-27T15:59:08.0105047Z ----------------------------------------------------------------------
2022-09-27T15:59:08.0175286Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.007s)
2022-09-27T15:59:08.0175890Z 
2022-09-27T15:59:08.0176201Z ----------------------------------------------------------------------
2022-09-27T15:59:08.0176538Z Ran 1 test in 0.007s
2022-09-27T15:59:08.0176711Z 
2022-09-27T15:59:08.0176825Z OK (skipped=1)
2022-09-27T15:59:08.0176987Z 
2022-09-27T15:59:08.0177118Z Generating XML reports...
2022-09-27T15:59:08.0229712Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155908.xml
2022-09-27T15:59:08.4671303Z ok (4.568s)
2022-09-27T15:59:08.4671691Z 
2022-09-27T15:59:08.4672463Z ----------------------------------------------------------------------
2022-09-27T15:59:08.4673087Z Ran 1 test in 4.569s
2022-09-27T15:59:08.4673262Z 
2022-09-27T15:59:08.4673358Z OK
2022-09-27T15:59:08.4673496Z 
2022-09-27T15:59:08.4673632Z Generating XML reports...
2022-09-27T15:59:08.4707980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155903.xml
2022-09-27T15:59:08.8772069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:08.8772612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:08.8773876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:08.8774702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:09.4290291Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:59:09.4311669Z 
2022-09-27T15:59:09.4312352Z Running tests...
2022-09-27T15:59:09.4313235Z ----------------------------------------------------------------------
2022-09-27T15:59:10.0968288Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:10.0968916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:10.0970375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:10.0970888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:10.5498347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:10.5498874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:10.5501127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:10.5501656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:10.6034250Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:10.6058024Z 
2022-09-27T15:59:10.6058336Z Running tests...
2022-09-27T15:59:10.6059411Z ----------------------------------------------------------------------
2022-09-27T15:59:10.6095820Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.004s)
2022-09-27T15:59:10.6096522Z 
2022-09-27T15:59:10.6097078Z ----------------------------------------------------------------------
2022-09-27T15:59:10.6097706Z Ran 1 test in 0.004s
2022-09-27T15:59:10.6098010Z 
2022-09-27T15:59:10.6098224Z OK (skipped=1)
2022-09-27T15:59:10.6098526Z 
2022-09-27T15:59:10.6098764Z Generating XML reports...
2022-09-27T15:59:10.6133836Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155910.xml
2022-09-27T15:59:11.0350215Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:11.0546426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27163
2022-09-27T15:59:11.0551198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27164
2022-09-27T15:59:11.0699678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:11.0724508Z 
2022-09-27T15:59:11.0725054Z Running tests...
2022-09-27T15:59:11.0725548Z ----------------------------------------------------------------------
2022-09-27T15:59:12.6505825Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:12.6507039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:12.6508251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:12.6509200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:12.6900222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:12.6901239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:12.6902444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:12.6903354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:12.7165467Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:12.7194657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:12.7195603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:12.7197870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:12.7198813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:12.7363763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27263
2022-09-27T15:59:12.7369296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27264
2022-09-27T15:59:13.1586518Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:13.1610010Z 
2022-09-27T15:59:13.1610517Z Running tests...
2022-09-27T15:59:13.1611027Z ----------------------------------------------------------------------
2022-09-27T15:59:13.1936190Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:13.2250200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:13.2251025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:13.2342446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:13.2349643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:13.3263931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:14.4392996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:14.4393523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:14.4394501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:14.4395052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:14.4551063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:14.4551912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:14.4553215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:14.4553703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:14.6341702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsml4veaa
2022-09-27T15:59:14.6342310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsml4veaa/_remote_module_non_scriptable.py
2022-09-27T15:59:14.7434340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ugeuh2e
2022-09-27T15:59:14.7435350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ugeuh2e/_remote_module_non_scriptable.py
2022-09-27T15:59:14.8623912Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:14.8828287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27346
2022-09-27T15:59:14.8833549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27347
2022-09-27T15:59:15.0266806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:15.0267365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:15.0268149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:15.0268860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:15.0375407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:15.1281339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:15.2593296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:15.2593835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:15.6450390Z ok (4.572s)
2022-09-27T15:59:15.6450757Z 
2022-09-27T15:59:15.6451413Z ----------------------------------------------------------------------
2022-09-27T15:59:15.6451896Z Ran 1 test in 4.572s
2022-09-27T15:59:15.6452077Z 
2022-09-27T15:59:15.6452175Z OK
2022-09-27T15:59:15.6452313Z 
2022-09-27T15:59:15.6452460Z Generating XML reports...
2022-09-27T15:59:15.6489240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155911.xml
2022-09-27T15:59:15.8671171Z ok (6.436s)
2022-09-27T15:59:15.8671562Z 
2022-09-27T15:59:15.8671971Z ----------------------------------------------------------------------
2022-09-27T15:59:15.8672620Z Ran 1 test in 6.436s
2022-09-27T15:59:15.8672811Z 
2022-09-27T15:59:15.8672906Z OK
2022-09-27T15:59:15.8673042Z 
2022-09-27T15:59:15.8673178Z Generating XML reports...
2022-09-27T15:59:15.8724222Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155909.xml
2022-09-27T15:59:16.5436135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:16.5436648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:16.5437825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:16.5438328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:16.5625625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:16.5626132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:16.5629277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:16.5630053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:17.1214928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:17.1829396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:17.1830206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:17.1924049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:17.1930948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:17.2032425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcrumot0
2022-09-27T15:59:17.2035416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcrumot0/_remote_module_non_scriptable.py
2022-09-27T15:59:17.2841241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:17.2941611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34kkqdfp
2022-09-27T15:59:17.2944461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34kkqdfp/_remote_module_non_scriptable.py
2022-09-27T15:59:17.8361562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:17.8362068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:17.8365488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:17.8366007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:18.0129681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:18.0130291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:18.0131760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:18.0132248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:18.3548969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:18.3573241Z 
2022-09-27T15:59:18.3573493Z Running tests...
2022-09-27T15:59:18.3573943Z ----------------------------------------------------------------------
2022-09-27T15:59:18.3598370Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:59:18.3600655Z 
2022-09-27T15:59:18.3601540Z ----------------------------------------------------------------------
2022-09-27T15:59:18.3601917Z Ran 1 test in 0.003s
2022-09-27T15:59:18.3602087Z 
2022-09-27T15:59:18.3602208Z OK (skipped=1)
2022-09-27T15:59:18.3602376Z 
2022-09-27T15:59:18.3602487Z Generating XML reports...
2022-09-27T15:59:18.3636234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155918.xml
2022-09-27T15:59:18.5301449Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:59:18.5324847Z 
2022-09-27T15:59:18.5325155Z Running tests...
2022-09-27T15:59:18.5325605Z ----------------------------------------------------------------------
2022-09-27T15:59:18.6785772Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... [1664294358.678074] [28fd6574525b:27347:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:18.6786411Z [1664294358.678074] [28fd6574525b:27347:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:18.6998471Z [1664294358.699431] [28fd6574525b:27346:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:18.6999265Z [1664294358.699431] [28fd6574525b:27346:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:19.0358480Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0360534Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0362410Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T15:59:19.0363424Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T15:59:19.0364864Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T15:59:19.0365875Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T15:59:19.0366470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:19.0367106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:19.0368548Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0370104Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0371902Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0373407Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0376729Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0378241Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0381327Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0382842Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0385381Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.0386878Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T15:59:19.5951716Z ok (6.433s)
2022-09-27T15:59:19.5951969Z 
2022-09-27T15:59:19.5952868Z ----------------------------------------------------------------------
2022-09-27T15:59:19.5953539Z Ran 1 test in 6.434s
2022-09-27T15:59:19.5953842Z 
2022-09-27T15:59:19.5953954Z OK
2022-09-27T15:59:19.5954098Z 
2022-09-27T15:59:19.5954238Z Generating XML reports...
2022-09-27T15:59:19.6005688Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155913.xml
2022-09-27T15:59:20.1935666Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:20.2130849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27538
2022-09-27T15:59:20.2137245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27539
2022-09-27T15:59:20.3638443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:20.3639383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:20.3640861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:20.3641804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:20.9918906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:20.9940927Z 
2022-09-27T15:59:20.9941396Z Running tests...
2022-09-27T15:59:20.9941930Z ----------------------------------------------------------------------
2022-09-27T15:59:20.9970185Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T15:59:20.9970551Z 
2022-09-27T15:59:20.9971053Z ----------------------------------------------------------------------
2022-09-27T15:59:20.9971385Z Ran 1 test in 0.003s
2022-09-27T15:59:20.9971555Z 
2022-09-27T15:59:20.9971676Z OK (skipped=1)
2022-09-27T15:59:20.9971871Z 
2022-09-27T15:59:20.9972003Z Generating XML reports...
2022-09-27T15:59:21.0005202Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155920.xml
2022-09-27T15:59:21.8202837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:21.8203367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:21.8205884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:21.8206394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:21.8794848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:21.8795326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:21.8797901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:21.8798400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:21.9001623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:21.9002094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:21.9004613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:21.9005101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:22.3855405Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:22.3878306Z 
2022-09-27T15:59:22.3878867Z Running tests...
2022-09-27T15:59:22.3879728Z ----------------------------------------------------------------------
2022-09-27T15:59:22.3925181Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.005s)
2022-09-27T15:59:22.3925538Z 
2022-09-27T15:59:22.3925825Z ----------------------------------------------------------------------
2022-09-27T15:59:22.3926174Z Ran 1 test in 0.005s
2022-09-27T15:59:22.3926343Z 
2022-09-27T15:59:22.3926457Z OK (skipped=1)
2022-09-27T15:59:22.3926619Z 
2022-09-27T15:59:22.3926728Z Generating XML reports...
2022-09-27T15:59:22.3961693Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155922.xml
2022-09-27T15:59:22.4519347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:22.4761365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:22.4762221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:22.4824834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:22.4831639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:22.5775468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:22.8822858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:22.8823384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:22.8824725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:22.8825223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:23.4156073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:23.4194840Z 
2022-09-27T15:59:23.4195285Z Running tests...
2022-09-27T15:59:23.4196351Z ----------------------------------------------------------------------
2022-09-27T15:59:23.4238847Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.004s)
2022-09-27T15:59:23.4239610Z 
2022-09-27T15:59:23.4240844Z ----------------------------------------------------------------------
2022-09-27T15:59:23.4241682Z Ran 1 test in 0.004s
2022-09-27T15:59:23.4242079Z 
2022-09-27T15:59:23.4242340Z OK (skipped=1)
2022-09-27T15:59:23.4242683Z 
2022-09-27T15:59:23.4242991Z Generating XML reports...
2022-09-27T15:59:23.4287740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155923.xml
2022-09-27T15:59:23.8762345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xdyouk4
2022-09-27T15:59:23.8763343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xdyouk4/_remote_module_non_scriptable.py
2022-09-27T15:59:23.9578714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_uxqdbf4
2022-09-27T15:59:23.9579686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_uxqdbf4/_remote_module_non_scriptable.py
2022-09-27T15:59:24.3772306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:24.3773069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:24.3774185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:24.3774668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:24.4726373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:24.4727666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:24.4736032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:24.4737240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:24.9250724Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:24.9275041Z 
2022-09-27T15:59:24.9275380Z Running tests...
2022-09-27T15:59:24.9276752Z ----------------------------------------------------------------------
2022-09-27T15:59:24.9308132Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-09-27T15:59:24.9308804Z 
2022-09-27T15:59:24.9309618Z ----------------------------------------------------------------------
2022-09-27T15:59:24.9310263Z Ran 1 test in 0.003s
2022-09-27T15:59:24.9310558Z 
2022-09-27T15:59:24.9310749Z OK (skipped=1)
2022-09-27T15:59:24.9311043Z 
2022-09-27T15:59:24.9311249Z Generating XML reports...
2022-09-27T15:59:24.9345817Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155924.xml
2022-09-27T15:59:25.0250103Z ok (6.492s)
2022-09-27T15:59:25.0250330Z 
2022-09-27T15:59:25.0250712Z ----------------------------------------------------------------------
2022-09-27T15:59:25.0251036Z Ran 1 test in 6.492s
2022-09-27T15:59:25.0251207Z 
2022-09-27T15:59:25.0251305Z OK
2022-09-27T15:59:25.0251448Z 
2022-09-27T15:59:25.0251612Z Generating XML reports...
2022-09-27T15:59:25.0287431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155918.xml
2022-09-27T15:59:25.4428077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:25.4429042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:25.4430263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:25.4431228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:25.9899762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:25.9921621Z 
2022-09-27T15:59:25.9922023Z Running tests...
2022-09-27T15:59:25.9922537Z ----------------------------------------------------------------------
2022-09-27T15:59:25.9944015Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:59:25.9944456Z 
2022-09-27T15:59:25.9944920Z ----------------------------------------------------------------------
2022-09-27T15:59:25.9945260Z Ran 1 test in 0.002s
2022-09-27T15:59:25.9945432Z 
2022-09-27T15:59:25.9945546Z OK (skipped=1)
2022-09-27T15:59:25.9945705Z 
2022-09-27T15:59:25.9945834Z Generating XML reports...
2022-09-27T15:59:25.9978673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155925.xml
2022-09-27T15:59:27.0021892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:27.0023147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:27.0028004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:27.0029194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:27.2113663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:27.2114218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:27.2116766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:27.2117277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:27.5220098Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:27.5243793Z 
2022-09-27T15:59:27.5244115Z Running tests...
2022-09-27T15:59:27.5244578Z ----------------------------------------------------------------------
2022-09-27T15:59:27.6898312Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:59:27.6919643Z 
2022-09-27T15:59:27.6919841Z Running tests...
2022-09-27T15:59:27.6920277Z ----------------------------------------------------------------------
2022-09-27T15:59:27.9439668Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:27.9440547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:27.9441590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:27.9442067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:28.4570779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:28.4593759Z 
2022-09-27T15:59:28.4594037Z Running tests...
2022-09-27T15:59:28.4594481Z ----------------------------------------------------------------------
2022-09-27T15:59:28.4613987Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:59:28.4614815Z 
2022-09-27T15:59:28.4615126Z ----------------------------------------------------------------------
2022-09-27T15:59:28.4615487Z Ran 1 test in 0.002s
2022-09-27T15:59:28.4615636Z 
2022-09-27T15:59:28.4615751Z OK (skipped=1)
2022-09-27T15:59:28.4615912Z 
2022-09-27T15:59:28.4616045Z Generating XML reports...
2022-09-27T15:59:28.4650591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155928.xml
2022-09-27T15:59:29.1219454Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:29.1420523Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27858
2022-09-27T15:59:29.1425524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27859
2022-09-27T15:59:29.2347061Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:29.2541606Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27861
2022-09-27T15:59:29.2547104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27862
2022-09-27T15:59:30.6221813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.6222343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.6223606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.6224108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.9154787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.9155294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.9156775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.9157268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:30.9298644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:30.9299100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:30.9302784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:30.9303280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:31.0241334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:31.0241837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:31.0243425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:31.0243943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:31.0547958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:31.0548447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:31.0551089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:31.0551582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:31.1128801Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:31.1151174Z 
2022-09-27T15:59:31.1151565Z Running tests...
2022-09-27T15:59:31.1152012Z ----------------------------------------------------------------------
2022-09-27T15:59:31.1175882Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:59:31.1176223Z 
2022-09-27T15:59:31.1176510Z ----------------------------------------------------------------------
2022-09-27T15:59:31.1177122Z Ran 1 test in 0.002s
2022-09-27T15:59:31.1177292Z 
2022-09-27T15:59:31.1177386Z OK (skipped=1)
2022-09-27T15:59:31.1177543Z 
2022-09-27T15:59:31.1177673Z Generating XML reports...
2022-09-27T15:59:31.1212051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155931.xml
2022-09-27T15:59:31.4309139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:31.4410588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:31.4411400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:31.4511037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:31.4517342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:31.5320523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:31.5423105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:31.5792942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:31.5793786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:31.5827387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:31.5835184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:31.6808059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:32.8507669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18vt00__
2022-09-27T15:59:32.8508300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18vt00__/_remote_module_non_scriptable.py
2022-09-27T15:59:32.9738222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z9qhut3
2022-09-27T15:59:32.9738813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z9qhut3/_remote_module_non_scriptable.py
2022-09-27T15:59:32.9909536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgjl_f8j
2022-09-27T15:59:32.9910089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgjl_f8j/_remote_module_non_scriptable.py
2022-09-27T15:59:32.9994307Z [1664294372.999077] [28fd6574525b:27858:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:32.9995024Z [1664294372.999077] [28fd6574525b:27858:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:33.0074369Z [1664294373.007073] [28fd6574525b:27859:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:33.0074881Z [1664294373.007073] [28fd6574525b:27859:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:33.0826007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u4igg64
2022-09-27T15:59:33.0826596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u4igg64/_remote_module_non_scriptable.py
2022-09-27T15:59:33.1043897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:33.1044381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:33.1048500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:33.1049000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:33.5981098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.5981668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.5991119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.5992069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.6119509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.6120058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.6128703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.6130952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.6665912Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:33.6691742Z 
2022-09-27T15:59:33.6692149Z Running tests...
2022-09-27T15:59:33.6692687Z ----------------------------------------------------------------------
2022-09-27T15:59:33.6719045Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T15:59:33.6719574Z 
2022-09-27T15:59:33.6719928Z ----------------------------------------------------------------------
2022-09-27T15:59:33.6720597Z Ran 1 test in 0.003s
2022-09-27T15:59:33.6720772Z 
2022-09-27T15:59:33.6720866Z OK (skipped=1)
2022-09-27T15:59:33.6721032Z 
2022-09-27T15:59:33.6721159Z Generating XML reports...
2022-09-27T15:59:33.6758372Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155933.xml
2022-09-27T15:59:33.7825653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.7826211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:33.8011223Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp5z9qhut3'>
2022-09-27T15:59:33.8011737Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T15:59:33.8012337Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp18vt00__'>
2022-09-27T15:59:33.8012797Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T15:59:34.1655403Z ok (6.473s)
2022-09-27T15:59:34.1655601Z 
2022-09-27T15:59:34.1655989Z ----------------------------------------------------------------------
2022-09-27T15:59:34.1656352Z Ran 1 test in 6.473s
2022-09-27T15:59:34.1656526Z 
2022-09-27T15:59:34.1656629Z OK
2022-09-27T15:59:34.1656769Z 
2022-09-27T15:59:34.1657147Z Generating XML reports...
2022-09-27T15:59:34.1691599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155927.xml
2022-09-27T15:59:34.4542512Z ok (6.929s)
2022-09-27T15:59:34.4542731Z 
2022-09-27T15:59:34.4543124Z ----------------------------------------------------------------------
2022-09-27T15:59:34.4543450Z Ran 1 test in 6.930s
2022-09-27T15:59:34.4543621Z 
2022-09-27T15:59:34.4543725Z OK
2022-09-27T15:59:34.4543866Z 
2022-09-27T15:59:34.4544005Z Generating XML reports...
2022-09-27T15:59:34.4579324Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155927.xml
2022-09-27T15:59:35.7649786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:35.7650300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:35.7651736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:35.7652237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:36.2520684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:36.2542612Z 
2022-09-27T15:59:36.2542846Z Running tests...
2022-09-27T15:59:36.2543359Z ----------------------------------------------------------------------
2022-09-27T15:59:36.2564100Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T15:59:36.2564425Z 
2022-09-27T15:59:36.2564713Z ----------------------------------------------------------------------
2022-09-27T15:59:36.2565050Z Ran 1 test in 0.002s
2022-09-27T15:59:36.2565217Z 
2022-09-27T15:59:36.2565330Z OK (skipped=1)
2022-09-27T15:59:36.2565493Z 
2022-09-27T15:59:36.2565619Z Generating XML reports...
2022-09-27T15:59:36.2599294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155936.xml
2022-09-27T15:59:36.3235583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:36.3236108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:36.3238645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:36.3239125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:36.5815964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:36.5816481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:36.5818711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:36.5819209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:36.8394961Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:59:36.8420273Z 
2022-09-27T15:59:36.8420496Z Running tests...
2022-09-27T15:59:36.8420941Z ----------------------------------------------------------------------
2022-09-27T15:59:37.0653135Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:37.0674802Z 
2022-09-27T15:59:37.0675106Z Running tests...
2022-09-27T15:59:37.0675548Z ----------------------------------------------------------------------
2022-09-27T15:59:38.1417467Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:38.1418361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:38.1419401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:38.1419916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:38.4823714Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:38.5025465Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28227
2022-09-27T15:59:38.5032106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28228
2022-09-27T15:59:38.6602116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:38.6625423Z 
2022-09-27T15:59:38.6625832Z Running tests...
2022-09-27T15:59:38.6626342Z ----------------------------------------------------------------------
2022-09-27T15:59:38.6762475Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:38.6956705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28230
2022-09-27T15:59:38.6962806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28231
2022-09-27T15:59:40.1769141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:40.1769680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:40.1770975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:40.1771486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:40.1898743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:40.1899228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:40.1903053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:40.1903547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:40.3065100Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:40.3261547Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28360
2022-09-27T15:59:40.3267497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28361
2022-09-27T15:59:40.4150405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:40.4151085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:40.4152191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:40.4152685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:40.4163806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:40.4164288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:40.4167422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:40.4167924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:40.6869931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:40.7011709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:40.7012508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:40.7072705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:40.7079296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:40.8027514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:40.9327598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:40.9483388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:40.9484227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:40.9530800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:40.9537795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:41.0580450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:42.0577935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:42.0578863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:42.0582947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:42.0584321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:42.0960004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:42.0960750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:42.0961362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:42.0961857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:42.1512811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rhmcj6o
2022-09-27T15:59:42.1513453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rhmcj6o/_remote_module_non_scriptable.py
2022-09-27T15:59:42.2455842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbgnt57x
2022-09-27T15:59:42.2456476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbgnt57x/_remote_module_non_scriptable.py
2022-09-27T15:59:42.3897896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn59eclks
2022-09-27T15:59:42.3898516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn59eclks/_remote_module_non_scriptable.py
2022-09-27T15:59:42.4703492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg96bvrc
2022-09-27T15:59:42.4704095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg96bvrc/_remote_module_non_scriptable.py
2022-09-27T15:59:42.4986089Z [1664294382.498105] [28fd6574525b:28230:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:42.4986615Z [1664294382.498105] [28fd6574525b:28230:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:42.5067821Z [1664294382.506400] [28fd6574525b:28231:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:42.5068323Z [1664294382.506400] [28fd6574525b:28231:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:42.6469811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:42.6470352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:42.6471154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:42.6472092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:42.6478516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:42.6479245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:42.6579630Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6580497Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6592963Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6593376Z 
2022-09-27T15:59:42.6608939Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6615569Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6628226Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6628830Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6640349Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6649070Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6660392Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6661002Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6673387Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6675764Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6687457Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6688073Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6699515Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6703991Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6715237Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6715847Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6728747Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6731030Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6742474Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6743088Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6754673Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6759258Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6770995Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6771609Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6783701Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6786375Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6797450Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6798062Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6810100Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6814555Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6826081Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6826679Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6838679Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6841960Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6853322Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6854835Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6865146Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6869494Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6881085Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6882342Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6893989Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6896102Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:42.6907838Z STAGE:2022-09-27 15:59:42 28361:28361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.6909078Z STAGE:2022-09-27 15:59:42 28360:28360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:42.7753572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:42.7754670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:42.7761207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:42.7762036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:42.7876633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:42.7877743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:42.7883821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:42.7884838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:43.1344855Z ok (4.471s)
2022-09-27T15:59:43.1345257Z 
2022-09-27T15:59:43.1345940Z ----------------------------------------------------------------------
2022-09-27T15:59:43.1346613Z Ran 1 test in 4.472s
2022-09-27T15:59:43.1346944Z 
2022-09-27T15:59:43.1347100Z OK
2022-09-27T15:59:43.1347349Z 
2022-09-27T15:59:43.1347616Z Generating XML reports...
2022-09-27T15:59:43.1381955Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155938.xml
2022-09-27T15:59:43.3119162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:43.3119723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:43.3131905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:43.3132423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:43.4140048Z ok (6.572s)
2022-09-27T15:59:43.4140254Z 
2022-09-27T15:59:43.4141162Z ----------------------------------------------------------------------
2022-09-27T15:59:43.4141828Z Ran 1 test in 6.572s
2022-09-27T15:59:43.4142128Z 
2022-09-27T15:59:43.4142312Z OK
2022-09-27T15:59:43.4142561Z 
2022-09-27T15:59:43.4142788Z Generating XML reports...
2022-09-27T15:59:43.4184664Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155936.xml
2022-09-27T15:59:44.1093323Z ok (7.041s)
2022-09-27T15:59:44.1093541Z 
2022-09-27T15:59:44.1093955Z ----------------------------------------------------------------------
2022-09-27T15:59:44.1094311Z Ran 1 test in 7.042s
2022-09-27T15:59:44.1094481Z 
2022-09-27T15:59:44.1094582Z OK
2022-09-27T15:59:44.1094725Z 
2022-09-27T15:59:44.1094862Z Generating XML reports...
2022-09-27T15:59:44.1130559Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155937.xml
2022-09-27T15:59:45.3454073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:45.3455282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:45.3458961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:45.3462073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:45.6393554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:45.6394197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:45.6396581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:45.6397338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:45.8349549Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:45.8371264Z 
2022-09-27T15:59:45.8371479Z Running tests...
2022-09-27T15:59:45.8372079Z ----------------------------------------------------------------------
2022-09-27T15:59:46.1307929Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:59:46.1330321Z 
2022-09-27T15:59:46.1330794Z Running tests...
2022-09-27T15:59:46.1331275Z ----------------------------------------------------------------------
2022-09-27T15:59:46.2620982Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:46.2621588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:46.2624379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:46.2625129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:46.7764281Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:46.7788598Z 
2022-09-27T15:59:46.7788860Z Running tests...
2022-09-27T15:59:46.7789319Z ----------------------------------------------------------------------
2022-09-27T15:59:47.4461341Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:47.4629386Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2022-09-27T15:59:47.4630063Z 
2022-09-27T15:59:47.4630374Z ----------------------------------------------------------------------
2022-09-27T15:59:47.4630725Z Ran 1 test in 1.626s
2022-09-27T15:59:47.4630899Z 
2022-09-27T15:59:47.4630994Z OK (skipped=1)
2022-09-27T15:59:47.4631165Z 
2022-09-27T15:59:47.4631294Z Generating XML reports...
2022-09-27T15:59:47.4664410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155945.xml
2022-09-27T15:59:47.7109139Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:47.7311095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28579
2022-09-27T15:59:47.7316660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28580
2022-09-27T15:59:48.4432610Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:48.4785499Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28645
2022-09-27T15:59:48.4792988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28646
2022-09-27T15:59:49.4340472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:49.4341049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:49.4342687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:49.4343559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:49.5119837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:49.5121136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:49.5122440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:49.5123181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:49.7112174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:49.7112723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:49.7114158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:49.7114999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:49.9467663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:50.0207809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:50.0208934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:50.0278227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:50.0285829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:50.1221605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:50.2163247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:50.2187070Z 
2022-09-27T15:59:50.2187307Z Running tests...
2022-09-27T15:59:50.2187975Z ----------------------------------------------------------------------
2022-09-27T15:59:50.2361107Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:50.2362259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:50.2363428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:50.2363932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:50.2415187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:50.2415945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:50.2418734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:50.2419545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:50.7745383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:50.7756995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:50.7757843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:50.7847678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:50.7854111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:50.7857519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:59:50.8772285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:50.8774773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:59:50.8775611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:59:50.8875945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:59:51.4238238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujozni8s
2022-09-27T15:59:51.4238939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujozni8s/_remote_module_non_scriptable.py
2022-09-27T15:59:51.5169141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vyals3x
2022-09-27T15:59:51.5169753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vyals3x/_remote_module_non_scriptable.py
2022-09-27T15:59:51.9238495Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:51.9449504Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28762
2022-09-27T15:59:51.9454485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28763
2022-09-27T15:59:52.2555700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxvbspkw
2022-09-27T15:59:52.2556830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxvbspkw/_remote_module_non_scriptable.py
2022-09-27T15:59:52.2882413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgir0jmy
2022-09-27T15:59:52.2883254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgir0jmy/_remote_module_non_scriptable.py
2022-09-27T15:59:52.3155735Z [1664294392.315116] [28fd6574525b:28645:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:52.3156237Z [1664294392.315116] [28fd6574525b:28645:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:52.3215492Z [1664294392.321187] [28fd6574525b:28646:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T15:59:52.3215982Z [1664294392.321187] [28fd6574525b:28646:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T15:59:52.5975059Z <class 'torch.nn.parameter.Parameter'>
2022-09-27T15:59:53.2378496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:53.2379087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:53.2390658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:53.2391206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T15:59:53.2436007Z ok (7.110s)
2022-09-27T15:59:53.2436218Z 
2022-09-27T15:59:53.2436612Z ----------------------------------------------------------------------
2022-09-27T15:59:53.2436950Z Ran 1 test in 7.111s
2022-09-27T15:59:53.2437118Z 
2022-09-27T15:59:53.2437209Z OK
2022-09-27T15:59:53.2437346Z 
2022-09-27T15:59:53.2437461Z Generating XML reports...
2022-09-27T15:59:53.2475575Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155946.xml
2022-09-27T15:59:53.6393902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:53.6394434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:53.6396748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:53.6397244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:53.6544403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:53.6544897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:53.6547150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:53.6547644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:53.9922213Z ok (7.213s)
2022-09-27T15:59:53.9922466Z 
2022-09-27T15:59:53.9922852Z ----------------------------------------------------------------------
2022-09-27T15:59:53.9923196Z Ran 1 test in 7.213s
2022-09-27T15:59:53.9923363Z 
2022-09-27T15:59:53.9923457Z OK
2022-09-27T15:59:53.9923592Z 
2022-09-27T15:59:53.9923731Z Generating XML reports...
2022-09-27T15:59:53.9978477Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155946.xml
2022-09-27T15:59:54.2397644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:54.2398197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:54.2398993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:54.2399726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:54.2506567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T15:59:54.3437998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:54.3626413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T15:59:54.3629819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T15:59:54.3633653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:59:54.3643708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T15:59:54.3777239Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.3855481Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.3881145Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.3884219Z 
2022-09-27T15:59:54.3911067Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.3950799Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.3977204Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.3991628Z 
2022-09-27T15:59:54.4012930Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4036562Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4063429Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4082416Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4091726Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4121450Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4141121Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4161262Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4173680Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4186269Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4212977Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4249912Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4264032Z 
2022-09-27T15:59:54.4278496Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4309730Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4347475Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4373493Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4374146Z 
2022-09-27T15:59:54.4415627Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4431551Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4460896Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4464034Z 
2022-09-27T15:59:54.4505468Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4527483Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4551825Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4556032Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4569881Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4572384Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4584732Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4587355Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4599165Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4599581Z 
2022-09-27T15:59:54.4611930Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4614317Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4624884Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4625650Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4637284Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4641581Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T15:59:54.4650748Z STAGE:2022-09-27 15:59:54 28763:28763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.4651362Z STAGE:2022-09-27 15:59:54 28762:28762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T15:59:54.8525875Z ok (4.634s)
2022-09-27T15:59:54.8526105Z 
2022-09-27T15:59:54.8526516Z ----------------------------------------------------------------------
2022-09-27T15:59:54.8526839Z Ran 1 test in 4.634s
2022-09-27T15:59:54.8527010Z 
2022-09-27T15:59:54.8527108Z OK
2022-09-27T15:59:54.8527243Z 
2022-09-27T15:59:54.8527396Z Generating XML reports...
2022-09-27T15:59:54.8562263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155950.xml
2022-09-27T15:59:55.4157798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:55.4158291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:55.4160338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:55.4161043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:55.9003478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T15:59:55.9026178Z 
2022-09-27T15:59:55.9026550Z Running tests...
2022-09-27T15:59:55.9027000Z ----------------------------------------------------------------------
2022-09-27T15:59:56.2581964Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:56.2582592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:56.2584007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:56.2584481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:56.7544177Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T15:59:56.7566927Z 
2022-09-27T15:59:56.7567220Z Running tests...
2022-09-27T15:59:56.7567870Z ----------------------------------------------------------------------
2022-09-27T15:59:56.9442027Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:56.9442648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:56.9444973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:56.9445448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:57.4571250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T15:59:57.4594711Z 
2022-09-27T15:59:57.4595201Z Running tests...
2022-09-27T15:59:57.4595690Z ----------------------------------------------------------------------
2022-09-27T15:59:57.5308850Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:57.5509404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28967
2022-09-27T15:59:57.5515160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28968
2022-09-27T15:59:58.4066025Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:58.4264157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29032
2022-09-27T15:59:58.4269266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29033
2022-09-27T15:59:59.1817246Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T15:59:59.2135895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29097
2022-09-27T15:59:59.2143692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29098
2022-09-27T15:59:59.2425783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:59.2426826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:59.2428574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:59.2429600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:59.2721945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T15:59:59.2722423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T15:59:59.2725539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T15:59:59.2726041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T15:59:59.7501499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T15:59:59.7745013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T15:59:59.7745878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:59.7806607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T15:59:59.7814218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T15:59:59.8782495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:00.1529989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:00.1530504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:00.1532594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:00.1533098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:00.1824866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:00.1825389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:00.1826372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:00.1826853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:00.6986894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:00.7086392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:00.7087204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:00.7088249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:00.7096130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:00.7097021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:00.7098741Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:00:00.7099758Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:00:00.8702099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:00.8703107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:00.8704311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:00.8705291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:00.9113520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:00.9114532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:00.9115775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:00.9116700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:01.2336637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprllx5rg8
2022-09-27T16:00:01.2337861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprllx5rg8/_remote_module_non_scriptable.py
2022-09-27T16:00:01.2770375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp948li6dk
2022-09-27T16:00:01.2771549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp948li6dk/_remote_module_non_scriptable.py
2022-09-27T16:00:01.4803313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:01.4804306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:01.4805681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:01.4806967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:01.4914540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:01.5818119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:02.0216079Z skip: Skipped due to small world size. (4.562s)
2022-09-27T16:00:02.0216646Z 
2022-09-27T16:00:02.0217347Z ----------------------------------------------------------------------
2022-09-27T16:00:02.0217994Z Ran 1 test in 4.562s
2022-09-27T16:00:02.0218838Z 
2022-09-27T16:00:02.0219079Z OK (skipped=1)
2022-09-27T16:00:02.0219373Z 
2022-09-27T16:00:02.0219625Z Generating XML reports...
2022-09-27T16:00:02.0252581Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155957.xml
2022-09-27T16:00:02.0992969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyllhowty
2022-09-27T16:00:02.0994119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyllhowty/_remote_module_non_scriptable.py
2022-09-27T16:00:02.1265090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjj6z2q_
2022-09-27T16:00:02.1265980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjj6z2q_/_remote_module_non_scriptable.py
2022-09-27T16:00:02.1546360Z [1664294402.154061] [28fd6574525b:29032:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:00:02.1547368Z [1664294402.154061] [28fd6574525b:29032:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:00:02.1573319Z [1664294402.156937] [28fd6574525b:29033:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:00:02.1574817Z [1664294402.156937] [28fd6574525b:29033:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:00:03.0529863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.0530422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.0540667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.0541368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.0643105Z ok (7.161s)
2022-09-27T16:00:03.0643674Z 
2022-09-27T16:00:03.0644367Z ----------------------------------------------------------------------
2022-09-27T16:00:03.0644772Z Ran 1 test in 7.162s
2022-09-27T16:00:03.0644945Z 
2022-09-27T16:00:03.0645037Z OK
2022-09-27T16:00:03.0645185Z 
2022-09-27T16:00:03.0645327Z Generating XML reports...
2022-09-27T16:00:03.0680821Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155955.xml
2022-09-27T16:00:03.0762020Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:00:03.0768167Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:00:03.3894053Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:00:03.3894685Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:00:03.3952130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.3952658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.3964318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.3964823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.4168517Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:00:03.4169969Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:00:03.5695061Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:00:03.5703343Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:00:03.5761787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.5762383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.5773331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:03.5774001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:04.2774575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:04.2779230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:04.2780559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:04.2781546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:04.3398581Z ok (7.583s)
2022-09-27T16:00:04.3398963Z 
2022-09-27T16:00:04.3399558Z ----------------------------------------------------------------------
2022-09-27T16:00:04.3400606Z Ran 1 test in 7.583s
2022-09-27T16:00:04.3400963Z 
2022-09-27T16:00:04.3401144Z OK
2022-09-27T16:00:04.3401778Z 
2022-09-27T16:00:04.3402037Z Generating XML reports...
2022-09-27T16:00:04.3436825Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155956.xml
2022-09-27T16:00:04.7908777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:04.7934432Z 
2022-09-27T16:00:04.7935065Z Running tests...
2022-09-27T16:00:04.7935578Z ----------------------------------------------------------------------
2022-09-27T16:00:05.1868979Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:05.1870244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:05.1871466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:05.1872476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:05.6826224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:00:05.6848564Z 
2022-09-27T16:00:05.6849014Z Running tests...
2022-09-27T16:00:05.6849857Z ----------------------------------------------------------------------
2022-09-27T16:00:06.3677416Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:06.3678621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:06.3679830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:06.3681080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:06.4583155Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:06.4790183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29310
2022-09-27T16:00:06.4795535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29311
2022-09-27T16:00:06.8614695Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:06.8637582Z 
2022-09-27T16:00:06.8638130Z Running tests...
2022-09-27T16:00:06.8639002Z ----------------------------------------------------------------------
2022-09-27T16:00:07.3399050Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:07.3598038Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29376
2022-09-27T16:00:07.3604997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29377
2022-09-27T16:00:08.1345090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:08.1346104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:08.1347294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:08.1348226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:08.1737182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:08.1738190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:08.1739419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:08.1740355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:08.5538774Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:08.5731722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29443
2022-09-27T16:00:08.5738118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29444
2022-09-27T16:00:08.7189252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:08.7189807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:08.7190589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:08.7191301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:08.7297398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:08.8203793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:09.0028635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:09.0029543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:09.0031323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:09.0031824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:09.0282886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:09.0283759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:09.0285226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:09.0285714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:09.6024285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:09.6201588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:09.6203197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:09.6228182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:09.6234797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:09.7215165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:10.2645719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:10.2646563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:10.2648311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:10.2649185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:10.2829800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:10.2830812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:10.2832115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:10.2833133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:10.7903183Z ok (5.997s)
2022-09-27T16:00:10.7903402Z 
2022-09-27T16:00:10.7904463Z ----------------------------------------------------------------------
2022-09-27T16:00:10.7904903Z Ran 1 test in 5.997s
2022-09-27T16:00:10.7905081Z 
2022-09-27T16:00:10.7905178Z OK
2022-09-27T16:00:10.7905324Z 
2022-09-27T16:00:10.7905484Z Generating XML reports...
2022-09-27T16:00:10.7940872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160004.xml
2022-09-27T16:00:10.8037005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:10.8179108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:10.8179933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:10.8242598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:10.8249980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:10.8252872Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:00:10.9193209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:10.9194081Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:00:11.0248536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0f3zb80s
2022-09-27T16:00:11.0249652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0f3zb80s/_remote_module_non_scriptable.py
2022-09-27T16:00:11.1681851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaei7z63_
2022-09-27T16:00:11.1683026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaei7z63_/_remote_module_non_scriptable.py
2022-09-27T16:00:12.2222184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8s3z72ng
2022-09-27T16:00:12.2222791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8s3z72ng/_remote_module_non_scriptable.py
2022-09-27T16:00:12.3013590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv6z33mk
2022-09-27T16:00:12.3014189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv6z33mk/_remote_module_non_scriptable.py
2022-09-27T16:00:12.3292347Z [1664294412.328786] [28fd6574525b:29443:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:00:12.3292881Z [1664294412.328786] [28fd6574525b:29443:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:00:12.3379148Z [1664294412.337534] [28fd6574525b:29444:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:00:12.3379689Z [1664294412.337534] [28fd6574525b:29444:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:00:12.6791761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:12.6792348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.0457870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:13.0458390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:13.0461206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:13.0461714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:13.1010061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.1010609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.1020514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.1020994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.1027167Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:00:13.1027759Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:00:13.1053536Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:00:13.1054324Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:00:13.1055729Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:00:13.1056747Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:00:13.1057467Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:00:13.1058139Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:00:13.3745720Z ok (7.689s)
2022-09-27T16:00:13.3745979Z 
2022-09-27T16:00:13.3746663Z ----------------------------------------------------------------------
2022-09-27T16:00:13.3747066Z Ran 1 test in 7.690s
2022-09-27T16:00:13.3747252Z 
2022-09-27T16:00:13.3747349Z OK
2022-09-27T16:00:13.3747496Z 
2022-09-27T16:00:13.3747632Z Generating XML reports...
2022-09-27T16:00:13.3794564Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160005.xml
2022-09-27T16:00:13.4876531Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:00:13.4880983Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:00:13.4946717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.4947267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.4959365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.4959886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:13.4965967Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:00:13.4967035Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:00:13.4992816Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:00:13.4994977Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:00:13.4996472Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:00:13.4999317Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:00:13.5694507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:13.5718531Z 
2022-09-27T16:00:13.5718847Z Running tests...
2022-09-27T16:00:13.5719290Z ----------------------------------------------------------------------
2022-09-27T16:00:14.3865167Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... ok (7.522s)
2022-09-27T16:00:14.3865854Z 
2022-09-27T16:00:14.3866574Z ----------------------------------------------------------------------
2022-09-27T16:00:14.3866969Z Ran 1 test in 7.523s
2022-09-27T16:00:14.3867137Z 
2022-09-27T16:00:14.3867252Z OK
2022-09-27T16:00:14.3867389Z 
2022-09-27T16:00:14.3867522Z Generating XML reports...
2022-09-27T16:00:14.3902681Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160006.xml
2022-09-27T16:00:15.3200522Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:15.3366531Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.764s)
2022-09-27T16:00:15.3367127Z 
2022-09-27T16:00:15.3367410Z ----------------------------------------------------------------------
2022-09-27T16:00:15.3367745Z Ran 1 test in 1.765s
2022-09-27T16:00:15.3367892Z 
2022-09-27T16:00:15.3368004Z OK (skipped=1)
2022-09-27T16:00:15.3368165Z 
2022-09-27T16:00:15.3368292Z Generating XML reports...
2022-09-27T16:00:15.3402153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160013.xml
2022-09-27T16:00:15.4454852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:15.4455355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:15.4457960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:15.4458452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:15.9842879Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:00:15.9865426Z 
2022-09-27T16:00:15.9865956Z Running tests...
2022-09-27T16:00:15.9866468Z ----------------------------------------------------------------------
2022-09-27T16:00:16.6056529Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:16.6057171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:16.6058068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:16.6058541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:17.1093667Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:17.1116681Z 
2022-09-27T16:00:17.1117232Z Running tests...
2022-09-27T16:00:17.1117732Z ----------------------------------------------------------------------
2022-09-27T16:00:17.1137218Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: flaky on PyTorch CI: No such file or directory: '/tmp/checkpoint.pt' (0.002s)
2022-09-27T16:00:17.1137614Z 
2022-09-27T16:00:17.1137912Z ----------------------------------------------------------------------
2022-09-27T16:00:17.1138483Z Ran 1 test in 0.002s
2022-09-27T16:00:17.1138652Z 
2022-09-27T16:00:17.1138761Z OK (skipped=1)
2022-09-27T16:00:17.1139195Z 
2022-09-27T16:00:17.1139374Z Generating XML reports...
2022-09-27T16:00:17.1173828Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160017.xml
2022-09-27T16:00:17.5514585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:17.5515124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:17.5515951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:17.5516443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:17.5848623Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:17.6042488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29694
2022-09-27T16:00:17.6049145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29695
2022-09-27T16:00:18.0515160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:18.0539897Z 
2022-09-27T16:00:18.0540572Z Running tests...
2022-09-27T16:00:18.0541488Z ----------------------------------------------------------------------
2022-09-27T16:00:19.1030452Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:19.1031701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:19.1032802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:19.1033666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:19.3317420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:19.3317953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:19.3320162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:19.3320883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:19.4044751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:19.4045259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:19.4048402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:19.4048878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:19.6027095Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:19.6049204Z 
2022-09-27T16:00:19.6049508Z Running tests...
2022-09-27T16:00:19.6049965Z ----------------------------------------------------------------------
2022-09-27T16:00:19.6069928Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:19.6070368Z 
2022-09-27T16:00:19.6070692Z ----------------------------------------------------------------------
2022-09-27T16:00:19.6071027Z Ran 1 test in 0.002s
2022-09-27T16:00:19.6071205Z 
2022-09-27T16:00:19.6071318Z OK (skipped=1)
2022-09-27T16:00:19.6071474Z 
2022-09-27T16:00:19.6071585Z Generating XML reports...
2022-09-27T16:00:19.6105188Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160019.xml
2022-09-27T16:00:19.7570243Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:19.7744026Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.720s)
2022-09-27T16:00:19.7745669Z 
2022-09-27T16:00:19.7746210Z ----------------------------------------------------------------------
2022-09-27T16:00:19.7746867Z Ran 1 test in 1.720s
2022-09-27T16:00:19.7747185Z 
2022-09-27T16:00:19.7747390Z OK (skipped=1)
2022-09-27T16:00:19.7747684Z 
2022-09-27T16:00:19.7747926Z Generating XML reports...
2022-09-27T16:00:19.7782241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160018.xml
2022-09-27T16:00:19.8419219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:19.9293535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:19.9295320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:19.9331870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:19.9338806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:20.0307840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:21.3211186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixassnbh
2022-09-27T16:00:21.3212240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixassnbh/_remote_module_non_scriptable.py
2022-09-27T16:00:21.4007174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7rvxcqu
2022-09-27T16:00:21.4008499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7rvxcqu/_remote_module_non_scriptable.py
2022-09-27T16:00:21.5488567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:21.5489090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:21.5491479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:21.5491961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:21.9210442Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:00:21.9418787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:21.9419260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:21.9422700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:21.9423198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:22.0560889Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:22.0584361Z 
2022-09-27T16:00:22.0584645Z Running tests...
2022-09-27T16:00:22.0585079Z ----------------------------------------------------------------------
2022-09-27T16:00:22.0606539Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:22.0607372Z 
2022-09-27T16:00:22.0607663Z ----------------------------------------------------------------------
2022-09-27T16:00:22.0607994Z Ran 1 test in 0.002s
2022-09-27T16:00:22.0608142Z 
2022-09-27T16:00:22.0608250Z OK (skipped=1)
2022-09-27T16:00:22.0608403Z 
2022-09-27T16:00:22.0608530Z Generating XML reports...
2022-09-27T16:00:22.0641901Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160022.xml
2022-09-27T16:00:22.4506686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:22.4528575Z 
2022-09-27T16:00:22.4529029Z Running tests...
2022-09-27T16:00:22.4529483Z ----------------------------------------------------------------------
2022-09-27T16:00:22.5162595Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... ok (6.529s)
2022-09-27T16:00:22.5162969Z 
2022-09-27T16:00:22.5163338Z ----------------------------------------------------------------------
2022-09-27T16:00:22.5163683Z Ran 1 test in 6.530s
2022-09-27T16:00:22.5163848Z 
2022-09-27T16:00:22.5163926Z OK
2022-09-27T16:00:22.5164063Z 
2022-09-27T16:00:22.5164190Z Generating XML reports...
2022-09-27T16:00:22.5199559Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160015.xml
2022-09-27T16:00:24.1413979Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:24.1574626Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.704s)
2022-09-27T16:00:24.1575577Z 
2022-09-27T16:00:24.1575888Z ----------------------------------------------------------------------
2022-09-27T16:00:24.1576232Z Ran 1 test in 1.705s
2022-09-27T16:00:24.1576396Z 
2022-09-27T16:00:24.1576489Z OK (skipped=1)
2022-09-27T16:00:24.1576654Z 
2022-09-27T16:00:24.1576783Z Generating XML reports...
2022-09-27T16:00:24.1609776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160022.xml
2022-09-27T16:00:24.1779254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:24.1779762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:24.1782583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:24.1783080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:24.6838518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:24.6839166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:24.6840819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:24.6841344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:24.6865096Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:24.6890807Z 
2022-09-27T16:00:24.6891012Z Running tests...
2022-09-27T16:00:24.6891731Z ----------------------------------------------------------------------
2022-09-27T16:00:24.6914200Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:24.6915050Z 
2022-09-27T16:00:24.6915343Z ----------------------------------------------------------------------
2022-09-27T16:00:24.6915682Z Ran 1 test in 0.002s
2022-09-27T16:00:24.6915845Z 
2022-09-27T16:00:24.6916205Z OK (skipped=1)
2022-09-27T16:00:24.6916368Z 
2022-09-27T16:00:24.6916478Z Generating XML reports...
2022-09-27T16:00:24.6951500Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160024.xml
2022-09-27T16:00:25.1786887Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:00:25.1808871Z 
2022-09-27T16:00:25.1809423Z Running tests...
2022-09-27T16:00:25.1809898Z ----------------------------------------------------------------------
2022-09-27T16:00:26.3363665Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:26.3364282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:26.3365986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:26.3366496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:26.6828193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:26.6828715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:26.6830053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:26.6830524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:26.7771754Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:26.7968796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30013
2022-09-27T16:00:26.7975061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30014
2022-09-27T16:00:26.8370795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:26.8394558Z 
2022-09-27T16:00:26.8394714Z Running tests...
2022-09-27T16:00:26.8395155Z ----------------------------------------------------------------------
2022-09-27T16:00:27.1862420Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:27.1885872Z 
2022-09-27T16:00:27.1886021Z Running tests...
2022-09-27T16:00:27.1886753Z ----------------------------------------------------------------------
2022-09-27T16:00:27.1907459Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:27.1908467Z 
2022-09-27T16:00:27.1909089Z ----------------------------------------------------------------------
2022-09-27T16:00:27.1909466Z Ran 1 test in 0.002s
2022-09-27T16:00:27.1909634Z 
2022-09-27T16:00:27.1909745Z OK (skipped=1)
2022-09-27T16:00:27.1909915Z 
2022-09-27T16:00:27.1910042Z Generating XML reports...
2022-09-27T16:00:27.1944355Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160027.xml
2022-09-27T16:00:28.4646583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:28.4647109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:28.4648132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:28.4648790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:28.4716205Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:28.4909426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:28.4910398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:28.4912917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:28.4913871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:28.4917349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30112
2022-09-27T16:00:28.4924801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30113
2022-09-27T16:00:28.9766494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:29.0081069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:29.0081953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:29.0172666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:29.0180669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:29.1241996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:29.2430728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:29.2431229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:29.2433435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:29.2433929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:29.7572784Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:29.7597178Z 
2022-09-27T16:00:29.7597490Z Running tests...
2022-09-27T16:00:29.7597932Z ----------------------------------------------------------------------
2022-09-27T16:00:29.7622090Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:29.7622628Z 
2022-09-27T16:00:29.7622959Z ----------------------------------------------------------------------
2022-09-27T16:00:29.7623296Z Ran 1 test in 0.002s
2022-09-27T16:00:29.7623459Z 
2022-09-27T16:00:29.7623569Z OK (skipped=1)
2022-09-27T16:00:29.7623726Z 
2022-09-27T16:00:29.7623835Z Generating XML reports...
2022-09-27T16:00:29.7659851Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160029.xml
2022-09-27T16:00:30.1406324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:30.1406881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:30.1408045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:30.1408550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:30.1455369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:30.1455844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:30.1459736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:30.1460242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:30.3756175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7q29art
2022-09-27T16:00:30.3757097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7q29art/_remote_module_non_scriptable.py
2022-09-27T16:00:30.4619931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpun3hnhi6
2022-09-27T16:00:30.4620794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpun3hnhi6/_remote_module_non_scriptable.py
2022-09-27T16:00:30.7078013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:30.7078555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:30.7079366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:30.7080112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:30.7087020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:30.7087508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:31.1019480Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:00:31.1021149Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:00:31.7081577Z ok (6.527s)
2022-09-27T16:00:31.7081828Z 
2022-09-27T16:00:31.7082217Z ----------------------------------------------------------------------
2022-09-27T16:00:31.7082558Z Ran 1 test in 6.527s
2022-09-27T16:00:31.7082729Z 
2022-09-27T16:00:31.7082822Z OK
2022-09-27T16:00:31.7082944Z 
2022-09-27T16:00:31.7083082Z Generating XML reports...
2022-09-27T16:00:31.7118172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160025.xml
2022-09-27T16:00:31.7785587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:31.7786389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:31.7787734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:31.7788235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:32.1799963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz29asp16
2022-09-27T16:00:32.1801117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz29asp16/_remote_module_non_scriptable.py
2022-09-27T16:00:32.1965777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2b_0paj6
2022-09-27T16:00:32.1968207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2b_0paj6/_remote_module_non_scriptable.py
2022-09-27T16:00:32.3150363Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:32.3173082Z 
2022-09-27T16:00:32.3173326Z Running tests...
2022-09-27T16:00:32.3173790Z ----------------------------------------------------------------------
2022-09-27T16:00:32.3196001Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:32.3196914Z 
2022-09-27T16:00:32.3197236Z ----------------------------------------------------------------------
2022-09-27T16:00:32.3197802Z Ran 1 test in 0.002s
2022-09-27T16:00:32.3197969Z 
2022-09-27T16:00:32.3198078Z OK (skipped=1)
2022-09-27T16:00:32.3198235Z 
2022-09-27T16:00:32.3198360Z Generating XML reports...
2022-09-27T16:00:32.3231571Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160032.xml
2022-09-27T16:00:32.6644054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:32.6644620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:33.2030203Z ok (6.363s)
2022-09-27T16:00:33.2030442Z 
2022-09-27T16:00:33.2030817Z ----------------------------------------------------------------------
2022-09-27T16:00:33.2031190Z Ran 1 test in 6.363s
2022-09-27T16:00:33.2031360Z 
2022-09-27T16:00:33.2031458Z OK
2022-09-27T16:00:33.2031594Z 
2022-09-27T16:00:33.2031739Z Generating XML reports...
2022-09-27T16:00:33.2068047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160026.xml
2022-09-27T16:00:33.9693792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:33.9694332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:33.9696341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:33.9696838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:34.4570279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:34.4570824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:34.4573312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:34.4573811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:34.5225321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:00:34.5248631Z 
2022-09-27T16:00:34.5249053Z Running tests...
2022-09-27T16:00:34.5249530Z ----------------------------------------------------------------------
2022-09-27T16:00:34.5271799Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-09-27T16:00:34.5272107Z 
2022-09-27T16:00:34.5272781Z ----------------------------------------------------------------------
2022-09-27T16:00:34.5273149Z Ran 1 test in 0.002s
2022-09-27T16:00:34.5273317Z 
2022-09-27T16:00:34.5273412Z OK (skipped=1)
2022-09-27T16:00:34.5273580Z 
2022-09-27T16:00:34.5273708Z Generating XML reports...
2022-09-27T16:00:34.5307506Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160034.xml
2022-09-27T16:00:34.9553730Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:34.9577250Z 
2022-09-27T16:00:34.9577501Z Running tests...
2022-09-27T16:00:34.9577942Z ----------------------------------------------------------------------
2022-09-27T16:00:34.9599930Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:34.9600775Z 
2022-09-27T16:00:34.9601103Z ----------------------------------------------------------------------
2022-09-27T16:00:34.9601441Z Ran 1 test in 0.002s
2022-09-27T16:00:34.9601612Z 
2022-09-27T16:00:34.9602001Z OK (skipped=1)
2022-09-27T16:00:34.9602146Z 
2022-09-27T16:00:34.9602280Z Generating XML reports...
2022-09-27T16:00:34.9637373Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160034.xml
2022-09-27T16:00:35.3420535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:35.3421058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:35.3422239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:35.3422730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:35.8964336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:35.8987216Z 
2022-09-27T16:00:35.8987723Z Running tests...
2022-09-27T16:00:35.8988253Z ----------------------------------------------------------------------
2022-09-27T16:00:36.4304597Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:36.4305798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:36.4306988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:36.4307964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:36.9199297Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:00:36.9221236Z 
2022-09-27T16:00:36.9221664Z Running tests...
2022-09-27T16:00:36.9222559Z ----------------------------------------------------------------------
2022-09-27T16:00:36.9347300Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:36.9348389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:36.9352094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:36.9353073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:37.4341533Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:37.4365650Z 
2022-09-27T16:00:37.4365911Z Running tests...
2022-09-27T16:00:37.4366605Z ----------------------------------------------------------------------
2022-09-27T16:00:37.4389467Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:37.4390398Z 
2022-09-27T16:00:37.4390699Z ----------------------------------------------------------------------
2022-09-27T16:00:37.4391059Z Ran 1 test in 0.002s
2022-09-27T16:00:37.4403388Z 
2022-09-27T16:00:37.4403655Z OK (skipped=1)
2022-09-27T16:00:37.4403937Z 
2022-09-27T16:00:37.4404078Z Generating XML reports...
2022-09-27T16:00:37.4434536Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160037.xml
2022-09-27T16:00:37.5607421Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:37.5815247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30414
2022-09-27T16:00:37.5820335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30415
2022-09-27T16:00:38.5776778Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:38.5975195Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30511
2022-09-27T16:00:38.5980882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30512
2022-09-27T16:00:39.2649791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:39.2650758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:39.2651920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:39.2652842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:39.2984652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:39.2985594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:39.2996255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:39.2997321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:39.5391928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:39.5392476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:39.5394478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:39.5394957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:39.8343446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:39.8343996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:39.8344803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:39.8345507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:39.8351256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:39.8351922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:40.0465762Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:40.0489599Z 
2022-09-27T16:00:40.0490003Z Running tests...
2022-09-27T16:00:40.0490499Z ----------------------------------------------------------------------
2022-09-27T16:00:40.0510874Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:40.0511961Z 
2022-09-27T16:00:40.0512566Z ----------------------------------------------------------------------
2022-09-27T16:00:40.0512911Z Ran 1 test in 0.002s
2022-09-27T16:00:40.0513090Z 
2022-09-27T16:00:40.0513216Z OK (skipped=1)
2022-09-27T16:00:40.0513379Z 
2022-09-27T16:00:40.0513510Z Generating XML reports...
2022-09-27T16:00:40.0547801Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160040.xml
2022-09-27T16:00:40.2187006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:40.2187512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:40.2189381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:40.2189860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:40.2697481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:40.2697996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:40.2700473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:40.2700987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:40.7547105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:40.8015638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:40.8016531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:40.8054447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:40.8061862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:40.9031158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:41.2311589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2wtaawy
2022-09-27T16:00:41.2312790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2wtaawy/_remote_module_non_scriptable.py
2022-09-27T16:00:41.2836407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4138q6o
2022-09-27T16:00:41.2837948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4138q6o/_remote_module_non_scriptable.py
2022-09-27T16:00:41.7373253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:41.7374324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:41.7387333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:41.7388367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:42.0928702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:42.0929749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:42.0930957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:42.0931917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:42.1866017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp394vknvn
2022-09-27T16:00:42.1867156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp394vknvn/_remote_module_non_scriptable.py
2022-09-27T16:00:42.2930214Z ok (6.394s)
2022-09-27T16:00:42.2930432Z 
2022-09-27T16:00:42.2931085Z ----------------------------------------------------------------------
2022-09-27T16:00:42.2931446Z Ran 1 test in 6.394s
2022-09-27T16:00:42.2931617Z 
2022-09-27T16:00:42.2931702Z OK
2022-09-27T16:00:42.2931838Z 
2022-09-27T16:00:42.2931984Z Generating XML reports...
2022-09-27T16:00:42.2966709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160035.xml
2022-09-27T16:00:42.3374968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7fo05jc
2022-09-27T16:00:42.3375998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7fo05jc/_remote_module_non_scriptable.py
2022-09-27T16:00:42.6293359Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:42.6317187Z 
2022-09-27T16:00:42.6317681Z Running tests...
2022-09-27T16:00:42.6318161Z ----------------------------------------------------------------------
2022-09-27T16:00:42.6339915Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:42.6341007Z 
2022-09-27T16:00:42.6341288Z ----------------------------------------------------------------------
2022-09-27T16:00:42.6341629Z Ran 1 test in 0.002s
2022-09-27T16:00:42.6341796Z 
2022-09-27T16:00:42.6341906Z OK (skipped=1)
2022-09-27T16:00:42.6342063Z 
2022-09-27T16:00:42.6342190Z Generating XML reports...
2022-09-27T16:00:42.6376010Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160042.xml
2022-09-27T16:00:43.9428614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:43.9429658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:44.4813872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:44.4814418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:44.4815833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:44.4816337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:44.5108871Z ok (7.589s)
2022-09-27T16:00:44.5109042Z 
2022-09-27T16:00:44.5109728Z ----------------------------------------------------------------------
2022-09-27T16:00:44.5110097Z Ran 1 test in 7.589s
2022-09-27T16:00:44.5110268Z 
2022-09-27T16:00:44.5110364Z OK
2022-09-27T16:00:44.5110483Z 
2022-09-27T16:00:44.5110620Z Generating XML reports...
2022-09-27T16:00:44.5146440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160036.xml
2022-09-27T16:00:44.6870816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:44.6871344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:44.6873903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:44.6874409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:44.9966449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:44.9989491Z 
2022-09-27T16:00:44.9989612Z Running tests...
2022-09-27T16:00:44.9990497Z ----------------------------------------------------------------------
2022-09-27T16:00:45.2003226Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:45.2027016Z 
2022-09-27T16:00:45.2027317Z Running tests...
2022-09-27T16:00:45.2028115Z ----------------------------------------------------------------------
2022-09-27T16:00:45.2049042Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:45.2049491Z 
2022-09-27T16:00:45.2049756Z ----------------------------------------------------------------------
2022-09-27T16:00:45.2050091Z Ran 1 test in 0.002s
2022-09-27T16:00:45.2050262Z 
2022-09-27T16:00:45.2050371Z OK (skipped=1)
2022-09-27T16:00:45.2050532Z 
2022-09-27T16:00:45.2050662Z Generating XML reports...
2022-09-27T16:00:45.2085995Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160045.xml
2022-09-27T16:00:46.6068488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:46.6069006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:46.6070728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:46.6071230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:46.6332592Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:46.6531407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30780
2022-09-27T16:00:46.6537948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30781
2022-09-27T16:00:47.0906004Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:00:47.0927461Z 
2022-09-27T16:00:47.0927764Z Running tests...
2022-09-27T16:00:47.0928201Z ----------------------------------------------------------------------
2022-09-27T16:00:47.1490220Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:47.1490841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:47.1492353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:47.1492841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:47.7272694Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:47.7295639Z 
2022-09-27T16:00:47.7296047Z Running tests...
2022-09-27T16:00:47.7296553Z ----------------------------------------------------------------------
2022-09-27T16:00:47.7317020Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:00:47.7317468Z 
2022-09-27T16:00:47.7317737Z ----------------------------------------------------------------------
2022-09-27T16:00:47.7318074Z Ran 1 test in 0.002s
2022-09-27T16:00:47.7318256Z 
2022-09-27T16:00:47.7318369Z OK (skipped=1)
2022-09-27T16:00:47.7318530Z 
2022-09-27T16:00:47.7318639Z Generating XML reports...
2022-09-27T16:00:47.7353395Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160047.xml
2022-09-27T16:00:48.2922879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:48.2923394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:48.2924922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:48.2925498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:48.3147480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:48.3148273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:48.3150591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:48.3151264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:48.7484698Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:48.7785998Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30881
2022-09-27T16:00:48.7791620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30882
2022-09-27T16:00:48.9083291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:48.9084097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:48.9084878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:48.9085602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:48.9191747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:49.0096692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:49.6835764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:49.6836739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:49.6837883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:49.6838848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:50.1966503Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:50.1991409Z 
2022-09-27T16:00:50.1991946Z Running tests...
2022-09-27T16:00:50.1992478Z ----------------------------------------------------------------------
2022-09-27T16:00:50.2011869Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:00:50.2012278Z 
2022-09-27T16:00:50.2012780Z ----------------------------------------------------------------------
2022-09-27T16:00:50.2013142Z Ran 1 test in 0.002s
2022-09-27T16:00:50.2013310Z 
2022-09-27T16:00:50.2013426Z OK (skipped=1)
2022-09-27T16:00:50.2013586Z 
2022-09-27T16:00:50.2013704Z Generating XML reports...
2022-09-27T16:00:50.2049026Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160050.xml
2022-09-27T16:00:50.3074244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0xiwji5
2022-09-27T16:00:50.3074941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0xiwji5/_remote_module_non_scriptable.py
2022-09-27T16:00:50.3076463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr_99ly3
2022-09-27T16:00:50.3079149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr_99ly3/_remote_module_non_scriptable.py
2022-09-27T16:00:50.3622867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:50.3623772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:50.3624899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:50.3625805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:50.4377987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:50.4379028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:50.4380878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:50.4382195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:50.7800763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:50.7801324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:50.7818718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:50.7819234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:50.7985608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:50.7986140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:50.8000523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:50.8001275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:00:50.8834752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:50.9729019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:50.9729859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:50.9747475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:50.9754777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:51.0743579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:51.3654885Z ok (6.366s)
2022-09-27T16:00:51.3655149Z 
2022-09-27T16:00:51.3656246Z ----------------------------------------------------------------------
2022-09-27T16:00:51.3656918Z Ran 1 test in 6.366s
2022-09-27T16:00:51.3657262Z 
2022-09-27T16:00:51.3657441Z OK
2022-09-27T16:00:51.3657720Z 
2022-09-27T16:00:51.3657998Z Generating XML reports...
2022-09-27T16:00:51.3693891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160044.xml
2022-09-27T16:00:52.2394814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:52.2395812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:52.2397039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:52.2398003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:52.4138579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xmh09sa
2022-09-27T16:00:52.4139722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xmh09sa/_remote_module_non_scriptable.py
2022-09-27T16:00:52.5110480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdeulqjo
2022-09-27T16:00:52.5111971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdeulqjo/_remote_module_non_scriptable.py
2022-09-27T16:00:52.8246665Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:00:52.8269695Z 
2022-09-27T16:00:52.8269997Z Running tests...
2022-09-27T16:00:52.8270449Z ----------------------------------------------------------------------
2022-09-27T16:00:53.4878153Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:53.4878756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:53.4881576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:53.4882101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:53.5466810Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-09-27T16:00:53.5469416Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-09-27T16:00:53.5485893Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-09-27T16:00:53.5487048Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-09-27T16:00:53.9999614Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:00:54.0023385Z 
2022-09-27T16:00:54.0023780Z Running tests...
2022-09-27T16:00:54.0024323Z ----------------------------------------------------------------------
2022-09-27T16:00:54.4683502Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:54.4889031Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31081
2022-09-27T16:00:54.4894160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31082
2022-09-27T16:00:54.6931966Z ok (7.600s)
2022-09-27T16:00:54.6932188Z 
2022-09-27T16:00:54.6932578Z ----------------------------------------------------------------------
2022-09-27T16:00:54.6932921Z Ran 1 test in 7.600s
2022-09-27T16:00:54.6933089Z 
2022-09-27T16:00:54.6933191Z OK
2022-09-27T16:00:54.6933330Z 
2022-09-27T16:00:54.6933466Z Generating XML reports...
2022-09-27T16:00:54.6968567Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160047.xml
2022-09-27T16:00:55.7522397Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:55.7721030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31147
2022-09-27T16:00:55.7726759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31148
2022-09-27T16:00:56.1624334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:56.1624898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:56.1625501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:56.1626002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:56.2409536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:56.2410075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:56.2412233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:56.2412726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:56.8224010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:56.8799048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:56.8799886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:56.8832177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:56.8838675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:56.9812304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:57.0555412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:57.0555921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:57.0557104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:57.0557789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:57.4684454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:57.4684996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:57.4687304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:57.4688123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:57.5495173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:00:57.5495690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:00:57.5497381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:00:57.5823663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:00:57.5824304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:00:57.5848071Z 
2022-09-27T16:00:57.5848701Z Running tests...
2022-09-27T16:00:57.5849246Z ----------------------------------------------------------------------
2022-09-27T16:00:58.0981745Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:00:58.0982400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:00:58.0983218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:58.0983932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:00:58.0990839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:00:58.0991319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:00:58.2748485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92jsgx61
2022-09-27T16:00:58.2749651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92jsgx61/_remote_module_non_scriptable.py
2022-09-27T16:00:58.3550161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi85ztf_j
2022-09-27T16:00:58.3551366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi85ztf_j/_remote_module_non_scriptable.py
2022-09-27T16:00:58.3824979Z [1664294458.381869] [28fd6574525b:31082:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:00:58.3825968Z [1664294458.381869] [28fd6574525b:31082:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:00:58.3831047Z [1664294458.382727] [28fd6574525b:31081:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:00:58.3832311Z [1664294458.382727] [28fd6574525b:31081:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:00:59.2893015Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:00:59.3100204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31272
2022-09-27T16:00:59.3104459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31273
2022-09-27T16:00:59.4517094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfp05t07
2022-09-27T16:00:59.4518264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfp05t07/_remote_module_non_scriptable.py
2022-09-27T16:00:59.4869518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpha7fb279
2022-09-27T16:00:59.4870145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpha7fb279/_remote_module_non_scriptable.py
2022-09-27T16:01:00.0109698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:00.0121535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:00.0156109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:00.0157274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:00.0462588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:00.0464284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:00.0489012Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:00.0490663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:00.2019559Z ok (7.375s)
2022-09-27T16:01:00.2019814Z 
2022-09-27T16:01:00.2020472Z ----------------------------------------------------------------------
2022-09-27T16:01:00.2021007Z Ran 1 test in 7.375s
2022-09-27T16:01:00.2021260Z 
2022-09-27T16:01:00.2021361Z OK
2022-09-27T16:01:00.2023603Z 
2022-09-27T16:01:00.2024149Z Generating XML reports...
2022-09-27T16:01:00.2059420Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160052.xml
2022-09-27T16:01:00.5833307Z ok (6.581s)
2022-09-27T16:01:00.5833568Z 
2022-09-27T16:01:00.5834212Z ----------------------------------------------------------------------
2022-09-27T16:01:00.5834572Z Ran 1 test in 6.581s
2022-09-27T16:01:00.5834743Z 
2022-09-27T16:01:00.5834839Z OK
2022-09-27T16:01:00.5834976Z 
2022-09-27T16:01:00.5835099Z Generating XML reports...
2022-09-27T16:01:00.5870964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160053.xml
2022-09-27T16:01:00.9415501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:00.9416019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:00.9418977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:00.9419458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:00.9865059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:00.9865557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:00.9867035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:00.9867501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:01.4897132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:01.5750973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:01.5752105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:01.5809169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:01.5816382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:01.6810857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:02.4173447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:02.4174171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:02.4175443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:02.4175949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:02.6938968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:02.6939525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:02.6941605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:02.6942438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:02.9403284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgtjmp29
2022-09-27T16:01:02.9404789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgtjmp29/_remote_module_non_scriptable.py
2022-09-27T16:01:02.9407465Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:02.9431889Z 
2022-09-27T16:01:02.9432575Z Running tests...
2022-09-27T16:01:02.9433267Z ----------------------------------------------------------------------
2022-09-27T16:01:03.0369690Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnos9k5z
2022-09-27T16:01:03.0370988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnos9k5z/_remote_module_non_scriptable.py
2022-09-27T16:01:03.1929060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:03.1951683Z 
2022-09-27T16:01:03.1952075Z Running tests...
2022-09-27T16:01:03.1952685Z ----------------------------------------------------------------------
2022-09-27T16:01:04.6309641Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:04.6510276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31429
2022-09-27T16:01:04.6516010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31430
2022-09-27T16:01:04.6593333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:04.6594314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:04.6779388Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqgtjmp29'>
2022-09-27T16:01:04.6779953Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:01:04.6780568Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpbnos9k5z'>
2022-09-27T16:01:04.6781015Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:01:04.8209699Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:04.8697929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31434
2022-09-27T16:01:04.8703367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31435
2022-09-27T16:01:05.3245206Z ok (7.739s)
2022-09-27T16:01:05.3246342Z 
2022-09-27T16:01:05.3248369Z ----------------------------------------------------------------------
2022-09-27T16:01:05.3249803Z Ran 1 test in 7.739s
2022-09-27T16:01:05.3250610Z 
2022-09-27T16:01:05.3250881Z OK
2022-09-27T16:01:05.3251220Z 
2022-09-27T16:01:05.3251485Z Generating XML reports...
2022-09-27T16:01:05.3304670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160057.xml
2022-09-27T16:01:06.4180224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:06.4180723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:06.4182150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:06.4182647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:06.4594611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:06.4595221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:06.4597164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:06.4597986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:06.7045821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:06.7046320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:06.7047756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:06.7048250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:06.7097131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:06.7097611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:06.7101000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:06.7101510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:06.9209898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:07.0147768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:07.0148643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:07.0222802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:07.0229855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:07.1160435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:07.2547573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:07.2548129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:07.2548936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:07.2549651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:07.2657913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:07.3563507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:07.5259992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:07.5260666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:07.5261707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:07.5262233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:08.0536224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:01:08.0559019Z 
2022-09-27T16:01:08.0559428Z Running tests...
2022-09-27T16:01:08.0559917Z ----------------------------------------------------------------------
2022-09-27T16:01:08.6112453Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3igh6ao
2022-09-27T16:01:08.6113179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3igh6ao/_remote_module_non_scriptable.py
2022-09-27T16:01:08.7558617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp223khswm
2022-09-27T16:01:08.7559856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp223khswm/_remote_module_non_scriptable.py
2022-09-27T16:01:08.7800064Z <class 'torch.nn.parameter.Parameter'>
2022-09-27T16:01:08.8422559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnhjfapt
2022-09-27T16:01:08.8423686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnhjfapt/_remote_module_non_scriptable.py
2022-09-27T16:01:08.9553767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs28avjn
2022-09-27T16:01:08.9554721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs28avjn/_remote_module_non_scriptable.py
2022-09-27T16:01:08.9909123Z [1664294468.990430] [28fd6574525b:31429:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:08.9909669Z [1664294468.990430] [28fd6574525b:31429:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:08.9915304Z [1664294468.991263] [28fd6574525b:31430:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:08.9915811Z [1664294468.991263] [28fd6574525b:31430:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:09.2819768Z ok (6.086s)
2022-09-27T16:01:09.2819982Z 
2022-09-27T16:01:09.2820340Z ----------------------------------------------------------------------
2022-09-27T16:01:09.2820680Z Ran 1 test in 6.087s
2022-09-27T16:01:09.2820847Z 
2022-09-27T16:01:09.2820941Z OK
2022-09-27T16:01:09.2821079Z 
2022-09-27T16:01:09.2821217Z Generating XML reports...
2022-09-27T16:01:09.2877232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160103.xml
2022-09-27T16:01:09.3819310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:09.3819857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:09.3916307Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqnhjfapt'>
2022-09-27T16:01:09.3916825Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:01:09.3917421Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpgs28avjn'>
2022-09-27T16:01:09.3917879Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:01:09.7275576Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:09.7475583Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31626
2022-09-27T16:01:09.7480883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31627
2022-09-27T16:01:09.9641466Z ok (7.021s)
2022-09-27T16:01:09.9641676Z 
2022-09-27T16:01:09.9642335Z ----------------------------------------------------------------------
2022-09-27T16:01:09.9642697Z Ran 1 test in 7.021s
2022-09-27T16:01:09.9642848Z 
2022-09-27T16:01:09.9642943Z OK
2022-09-27T16:01:09.9643079Z 
2022-09-27T16:01:09.9643230Z Generating XML reports...
2022-09-27T16:01:09.9679527Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160102.xml
2022-09-27T16:01:11.4729124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:11.4729758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:11.4731617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:11.4732114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:11.4964701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:11.4965181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:11.4968556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:11.4969397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:11.5633283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:11.5634211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:11.5635357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:11.5635847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:11.9853677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:12.0088940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:12.0089769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:12.0158112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:12.0164820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:12.0503674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:12.0526081Z 
2022-09-27T16:01:12.0526317Z Running tests...
2022-09-27T16:01:12.0526892Z ----------------------------------------------------------------------
2022-09-27T16:01:12.1104037Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:12.1195575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:12.1196063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:12.1199937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:12.1200793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:12.6999870Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:12.7025126Z 
2022-09-27T16:01:12.7025466Z Running tests...
2022-09-27T16:01:12.7025893Z ----------------------------------------------------------------------
2022-09-27T16:01:13.4006488Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oothn95
2022-09-27T16:01:13.4007831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oothn95/_remote_module_non_scriptable.py
2022-09-27T16:01:13.5302006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht55mlo_
2022-09-27T16:01:13.5302931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht55mlo_/_remote_module_non_scriptable.py
2022-09-27T16:01:13.7629045Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:13.7836536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31772
2022-09-27T16:01:13.7843648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31773
2022-09-27T16:01:14.3824971Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:14.4029323Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31775
2022-09-27T16:01:14.4034290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31776
2022-09-27T16:01:15.1918096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:15.1918684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:15.1932270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:15.1933116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:15.5279012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:15.5279588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:15.5280965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:15.5281473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:15.5399709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:15.5402244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:15.5402859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:15.5403353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:16.0616577Z ok (8.005s)
2022-09-27T16:01:16.0616925Z 
2022-09-27T16:01:16.0617386Z ----------------------------------------------------------------------
2022-09-27T16:01:16.0617722Z Ran 1 test in 8.006s
2022-09-27T16:01:16.0617896Z 
2022-09-27T16:01:16.0617997Z OK
2022-09-27T16:01:16.0618136Z 
2022-09-27T16:01:16.0618271Z Generating XML reports...
2022-09-27T16:01:16.0653253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160108.xml
2022-09-27T16:01:16.0927248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:16.0928008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:16.0928773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:16.0929468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:16.0942524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:16.0943003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:16.0946570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:16.0947055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:16.1036015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:16.1146896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:16.1147408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:16.1148905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:16.1149415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:16.1941913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:16.6315266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:16.6563079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:16.6563916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:16.6620451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:16.6627843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:16.6737997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dyl7lj7
2022-09-27T16:01:16.6739366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dyl7lj7/_remote_module_non_scriptable.py
2022-09-27T16:01:16.7576745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:16.7684020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5m73piwq
2022-09-27T16:01:16.7685506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5m73piwq/_remote_module_non_scriptable.py
2022-09-27T16:01:17.5329554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4ptbs3v
2022-09-27T16:01:17.5330202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4ptbs3v/_remote_module_non_scriptable.py
2022-09-27T16:01:17.5901136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdarnztqh
2022-09-27T16:01:17.5902069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdarnztqh/_remote_module_non_scriptable.py
2022-09-27T16:01:18.1946667Z ok (6.142s)
2022-09-27T16:01:18.1946875Z 
2022-09-27T16:01:18.1947287Z ----------------------------------------------------------------------
2022-09-27T16:01:18.1947611Z Ran 1 test in 6.142s
2022-09-27T16:01:18.1947781Z 
2022-09-27T16:01:18.1947879Z OK
2022-09-27T16:01:18.1948018Z 
2022-09-27T16:01:18.1948159Z Generating XML reports...
2022-09-27T16:01:18.1953977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:18.1954440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:18.1958417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:18.1958921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:18.1983308Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160112.xml
2022-09-27T16:01:18.2162877Z [1664294478.215848] [28fd6574525b:31776:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:18.2163427Z [1664294478.215848] [28fd6574525b:31776:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:18.2675496Z [1664294478.267111] [28fd6574525b:31775:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:18.2676015Z [1664294478.267111] [28fd6574525b:31775:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:18.5844700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:18.5845299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:18.7240557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:01:18.7262891Z 
2022-09-27T16:01:18.7263157Z Running tests...
2022-09-27T16:01:18.7263607Z ----------------------------------------------------------------------
2022-09-27T16:01:19.3156580Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... ok (6.612s)
2022-09-27T16:01:19.3157910Z 
2022-09-27T16:01:19.3159442Z ----------------------------------------------------------------------
2022-09-27T16:01:19.3160824Z Ran 1 test in 6.613s
2022-09-27T16:01:19.3161220Z 
2022-09-27T16:01:19.3161474Z OK
2022-09-27T16:01:19.3161789Z 
2022-09-27T16:01:19.3162097Z Generating XML reports...
2022-09-27T16:01:19.3219677Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160112.xml
2022-09-27T16:01:20.3475076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:20.3475586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:20.3478211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:20.3478710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:20.4419587Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:20.4783725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32067
2022-09-27T16:01:20.4871226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32068
2022-09-27T16:01:20.8872390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:20.8894574Z 
2022-09-27T16:01:20.8895105Z Running tests...
2022-09-27T16:01:20.8895643Z ----------------------------------------------------------------------
2022-09-27T16:01:21.5176544Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:21.5177164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:21.5178959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:21.5179435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:22.0357728Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:22.0381784Z 
2022-09-27T16:01:22.0382091Z Running tests...
2022-09-27T16:01:22.0382516Z ----------------------------------------------------------------------
2022-09-27T16:01:22.1556126Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:22.1556744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:22.1559363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:22.1559867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:22.1561289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:22.1561757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:22.1565935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:22.1566423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:22.5041702Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:22.5239496Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32137
2022-09-27T16:01:22.5244796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32138
2022-09-27T16:01:22.6863326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:22.6878865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:22.6879647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:22.6965784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:22.6972559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:22.6976357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:01:22.7893506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:22.7896289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:01:22.7897339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:01:22.7990931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:01:23.7273150Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:23.7480437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32214
2022-09-27T16:01:23.7486236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32215
2022-09-27T16:01:24.1930707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp824_sx5l
2022-09-27T16:01:24.1931560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp824_sx5l/_remote_module_non_scriptable.py
2022-09-27T16:01:24.2032681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:24.2033189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:24.2036500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:24.2037003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:24.2194541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxpjgotj
2022-09-27T16:01:24.2196125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxpjgotj/_remote_module_non_scriptable.py
2022-09-27T16:01:24.2428315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:24.2428800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:24.2431894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:24.2432404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:24.8655725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:24.8656281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:24.8657085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:24.8657783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:24.8764378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:24.9671336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:25.4439344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:25.4439876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:25.4440691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:25.4441168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:25.4555438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:25.4556038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:25.4557451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:25.4557926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:25.8715789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:25.8716817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:25.8726848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:25.8727413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:25.9913824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:25.9914409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:25.9915226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:25.9915926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:25.9921375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:25.9921876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:26.2397176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4itqx01
2022-09-27T16:01:26.2397789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4itqx01/_remote_module_non_scriptable.py
2022-09-27T16:01:26.3745294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3e0ubugm
2022-09-27T16:01:26.3747167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3e0ubugm/_remote_module_non_scriptable.py
2022-09-27T16:01:26.8005471Z ok (8.074s)
2022-09-27T16:01:26.8005827Z 
2022-09-27T16:01:26.8006459Z ----------------------------------------------------------------------
2022-09-27T16:01:26.8007066Z Ran 1 test in 8.074s
2022-09-27T16:01:26.8007378Z 
2022-09-27T16:01:26.8007566Z OK
2022-09-27T16:01:26.8007852Z 
2022-09-27T16:01:26.8008102Z Generating XML reports...
2022-09-27T16:01:26.8045786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160118.xml
2022-09-27T16:01:26.9473972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:26.9475068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:27.5353523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptelq7h6g
2022-09-27T16:01:27.5353963Z ok (6.646s)
2022-09-27T16:01:27.5354368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptelq7h6g/_remote_module_non_scriptable.py
2022-09-27T16:01:27.5354703Z 
2022-09-27T16:01:27.5355089Z ----------------------------------------------------------------------
2022-09-27T16:01:27.5355432Z Ran 1 test in 6.646s
2022-09-27T16:01:27.5355596Z 
2022-09-27T16:01:27.5355677Z OK
2022-09-27T16:01:27.5356049Z 
2022-09-27T16:01:27.5356194Z Generating XML reports...
2022-09-27T16:01:27.5392448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160120.xml
2022-09-27T16:01:27.5570353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6izp1moi
2022-09-27T16:01:27.5571766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6izp1moi/_remote_module_non_scriptable.py
2022-09-27T16:01:27.5829089Z [1664294487.582478] [28fd6574525b:32214:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:27.5829602Z [1664294487.582478] [28fd6574525b:32214:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:27.5923283Z [1664294487.591987] [28fd6574525b:32215:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:27.5923786Z [1664294487.591987] [28fd6574525b:32215:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:28.4512354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:28.4513222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:29.0601725Z ok (7.022s)
2022-09-27T16:01:29.0601932Z 
2022-09-27T16:01:29.0602334Z ----------------------------------------------------------------------
2022-09-27T16:01:29.0602678Z Ran 1 test in 7.022s
2022-09-27T16:01:29.0602848Z 
2022-09-27T16:01:29.0602942Z OK
2022-09-27T16:01:29.0603081Z 
2022-09-27T16:01:29.0603201Z Generating XML reports...
2022-09-27T16:01:29.0637493Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160122.xml
2022-09-27T16:01:29.1020124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:29.1020649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:29.1023227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:29.1023716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:29.6098704Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:01:29.6122117Z 
2022-09-27T16:01:29.6122356Z Running tests...
2022-09-27T16:01:29.6122817Z ----------------------------------------------------------------------
2022-09-27T16:01:29.6656537Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:29.6657133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:29.6659393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:29.6659901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:30.1615363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:30.1637771Z 
2022-09-27T16:01:30.1638294Z Running tests...
2022-09-27T16:01:30.1639364Z ----------------------------------------------------------------------
2022-09-27T16:01:31.0707549Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:31.0708164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:31.0708768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:31.0709254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:31.2875599Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:31.3076826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32427
2022-09-27T16:01:31.3082293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32428
2022-09-27T16:01:31.5726655Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:31.5749154Z 
2022-09-27T16:01:31.5749716Z Running tests...
2022-09-27T16:01:31.5750244Z ----------------------------------------------------------------------
2022-09-27T16:01:31.5772287Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:01:31.5772666Z 
2022-09-27T16:01:31.5772951Z ----------------------------------------------------------------------
2022-09-27T16:01:31.5773273Z Ran 1 test in 0.002s
2022-09-27T16:01:31.5773440Z 
2022-09-27T16:01:31.5773554Z OK (skipped=1)
2022-09-27T16:01:31.5773726Z 
2022-09-27T16:01:31.5775000Z Generating XML reports...
2022-09-27T16:01:31.5810088Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160131.xml
2022-09-27T16:01:31.7242454Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:31.7438233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32430
2022-09-27T16:01:31.7444166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32431
2022-09-27T16:01:33.0427088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:33.0427594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:33.0428766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:33.0429256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:33.1078821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:33.1079300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:33.1080977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:33.1081465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:33.5115802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:33.5116296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:33.5119137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:33.5119631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:33.5374283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:33.5374743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:33.5377246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:33.5377731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:33.5514283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:33.6257460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:33.6258249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:33.6328532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:33.6335479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:33.6337994Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:01:33.6932809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:33.6933302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:33.6935985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:33.6936453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:33.7269930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:33.7271140Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:01:34.0716591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:34.0717173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:34.0718206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:34.0718915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:34.0825117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:34.1731649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:34.2086207Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:34.2109897Z 
2022-09-27T16:01:34.2110214Z Running tests...
2022-09-27T16:01:34.2110947Z ----------------------------------------------------------------------
2022-09-27T16:01:34.2134378Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:01:34.2134751Z 
2022-09-27T16:01:34.2135029Z ----------------------------------------------------------------------
2022-09-27T16:01:34.2135352Z Ran 1 test in 0.002s
2022-09-27T16:01:34.2135516Z 
2022-09-27T16:01:34.2135635Z OK (skipped=1)
2022-09-27T16:01:34.2135795Z 
2022-09-27T16:01:34.2135924Z Generating XML reports...
2022-09-27T16:01:34.2171279Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160134.xml
2022-09-27T16:01:35.0094986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpth5_hhds
2022-09-27T16:01:35.0096217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpth5_hhds/_remote_module_non_scriptable.py
2022-09-27T16:01:35.1076759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjyrsj05
2022-09-27T16:01:35.1077382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjyrsj05/_remote_module_non_scriptable.py
2022-09-27T16:01:35.5151565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72q7568f
2022-09-27T16:01:35.5152164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72q7568f/_remote_module_non_scriptable.py
2022-09-27T16:01:35.5817154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8qikerl
2022-09-27T16:01:35.5818489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8qikerl/_remote_module_non_scriptable.py
2022-09-27T16:01:36.0258272Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:01:36.2802765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:36.2803288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:36.2804687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:36.2805185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:36.6554903Z ok (6.491s)
2022-09-27T16:01:36.6555294Z 
2022-09-27T16:01:36.6555991Z ----------------------------------------------------------------------
2022-09-27T16:01:36.6556617Z Ran 1 test in 6.492s
2022-09-27T16:01:36.6556934Z 
2022-09-27T16:01:36.6557109Z OK
2022-09-27T16:01:36.6557364Z 
2022-09-27T16:01:36.6557637Z Generating XML reports...
2022-09-27T16:01:36.6594414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160130.xml
2022-09-27T16:01:36.7041819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:36.7042891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:36.7052396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:36.7053352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:36.7264998Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:01:36.7266055Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:01:36.8143068Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:36.8165512Z 
2022-09-27T16:01:36.8166037Z Running tests...
2022-09-27T16:01:36.8166724Z ----------------------------------------------------------------------
2022-09-27T16:01:36.8184829Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:01:36.8185470Z 
2022-09-27T16:01:36.8185844Z ----------------------------------------------------------------------
2022-09-27T16:01:36.8186190Z Ran 1 test in 0.002s
2022-09-27T16:01:36.8186338Z 
2022-09-27T16:01:36.8186452Z OK (skipped=1)
2022-09-27T16:01:36.8186610Z 
2022-09-27T16:01:36.8186736Z Generating XML reports...
2022-09-27T16:01:36.8220814Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160136.xml
2022-09-27T16:01:36.9395731Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:01:36.9396332Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:01:36.9456504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:36.9457029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:36.9468040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:36.9468522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:36.9677708Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:01:36.9678311Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:01:37.1387723Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:01:37.1397292Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:01:37.1485876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:37.1486390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:37.1502254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:37.1502742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:38.1228627Z ok (8.510s)
2022-09-27T16:01:38.1228844Z 
2022-09-27T16:01:38.1229246Z ----------------------------------------------------------------------
2022-09-27T16:01:38.1229605Z Ran 1 test in 8.511s
2022-09-27T16:01:38.1229759Z 
2022-09-27T16:01:38.1229862Z OK
2022-09-27T16:01:38.1229997Z 
2022-09-27T16:01:38.1230155Z Generating XML reports...
2022-09-27T16:01:38.1265686Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160129.xml
2022-09-27T16:01:38.7497034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:38.7497553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:38.7500087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:38.7500598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:38.7507096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:38.7507563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:38.7511382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:38.7511883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:39.2737229Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:39.2778889Z 
2022-09-27T16:01:39.2780391Z Running tests...
2022-09-27T16:01:39.2781447Z ----------------------------------------------------------------------
2022-09-27T16:01:39.2822817Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.004s)
2022-09-27T16:01:39.2831723Z 
2022-09-27T16:01:39.2833081Z ----------------------------------------------------------------------
2022-09-27T16:01:39.2834365Z Ran 1 test in 0.004s
2022-09-27T16:01:39.2835451Z 
2022-09-27T16:01:39.2836187Z OK (skipped=1)
2022-09-27T16:01:39.2836619Z 
2022-09-27T16:01:39.2836925Z Generating XML reports...
2022-09-27T16:01:39.2886573Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160139.xml
2022-09-27T16:01:39.2960967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:39.3003708Z 
2022-09-27T16:01:39.3004797Z Running tests...
2022-09-27T16:01:39.3006400Z ----------------------------------------------------------------------
2022-09-27T16:01:40.2492553Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:40.2494049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:40.2497228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:40.2498383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:40.7954623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:01:40.7977772Z 
2022-09-27T16:01:40.7978203Z Running tests...
2022-09-27T16:01:40.7978659Z ----------------------------------------------------------------------
2022-09-27T16:01:40.8914143Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:40.9107860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32794
2022-09-27T16:01:40.9114595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32795
2022-09-27T16:01:41.1783202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:41.1783720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:41.1785749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:41.1786251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:41.7305626Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:41.7328287Z 
2022-09-27T16:01:41.7328743Z Running tests...
2022-09-27T16:01:41.7329196Z ----------------------------------------------------------------------
2022-09-27T16:01:41.7355593Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-09-27T16:01:41.7355946Z 
2022-09-27T16:01:41.7356225Z ----------------------------------------------------------------------
2022-09-27T16:01:41.7356559Z Ran 1 test in 0.003s
2022-09-27T16:01:41.7356724Z 
2022-09-27T16:01:41.7356834Z OK (skipped=1)
2022-09-27T16:01:41.7356974Z 
2022-09-27T16:01:41.7357101Z Generating XML reports...
2022-09-27T16:01:41.7390873Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160141.xml
2022-09-27T16:01:42.4244507Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:42.4449468Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32861
2022-09-27T16:01:42.4454883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32862
2022-09-27T16:01:42.5665253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:42.5665752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:42.5668300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:42.5668797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:42.5669389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:42.5669830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:42.5673346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:42.5673844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:43.1506794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:43.1511657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:43.1516969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:43.1532168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:43.1638072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:43.2561419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:43.7970323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:43.7970875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:43.7971728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:43.7972205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:44.1510122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:44.1510632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:44.1511750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:44.1512230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:44.1604380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:44.1604977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:44.1607716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:44.1608603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:44.3203810Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:44.3228002Z 
2022-09-27T16:01:44.3228393Z Running tests...
2022-09-27T16:01:44.3229015Z ----------------------------------------------------------------------
2022-09-27T16:01:44.5714023Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30gbgqdc
2022-09-27T16:01:44.5714734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30gbgqdc/_remote_module_non_scriptable.py
2022-09-27T16:01:44.5838020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_hmq78h
2022-09-27T16:01:44.5840826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_hmq78h/_remote_module_non_scriptable.py
2022-09-27T16:01:44.6783267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:44.6967379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:44.6986939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:44.6987638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:44.6994374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:44.6996946Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:01:44.7981437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:44.7982705Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:01:45.0639324Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:01:45.0641254Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:01:45.7223796Z ok (6.422s)
2022-09-27T16:01:45.7224060Z 
2022-09-27T16:01:45.7224477Z ----------------------------------------------------------------------
2022-09-27T16:01:45.7224826Z Ran 1 test in 6.422s
2022-09-27T16:01:45.7225003Z 
2022-09-27T16:01:45.7225100Z OK
2022-09-27T16:01:45.7225237Z 
2022-09-27T16:01:45.7225355Z Generating XML reports...
2022-09-27T16:01:45.7260532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160139.xml
2022-09-27T16:01:46.0995158Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:46.1197806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_10nxzi2
2022-09-27T16:01:46.1199004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_10nxzi2/_remote_module_non_scriptable.py
2022-09-27T16:01:46.1200374Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32985
2022-09-27T16:01:46.1207699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32986
2022-09-27T16:01:46.2059907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaet_ji7s
2022-09-27T16:01:46.2060979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaet_ji7s/_remote_module_non_scriptable.py
2022-09-27T16:01:47.8233892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:47.8234435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:47.8235347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:47.8235854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:47.8595720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:47.8596220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:47.8599007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:47.8599522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:47.9061722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:47.9062286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:47.9071942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:47.9072461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:47.9079139Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:01:47.9079729Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:01:47.9107051Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:01:47.9107975Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:01:47.9109620Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:01:47.9110311Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:01:47.9110979Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:01:47.9111629Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:01:47.9552581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:47.9553135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:47.9554137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:47.9554937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:48.2827335Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:01:48.2829455Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:01:48.2889984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:48.2890888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:48.2902267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:48.2903219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:48.2908112Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:01:48.2908709Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:01:48.2934696Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:01:48.2936065Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:01:48.2937310Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:01:48.2938576Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:01:48.3342719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:48.3899184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:48.3900021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:48.3951625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:48.3957715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:48.4409795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:48.4431111Z 
2022-09-27T16:01:48.4431519Z Running tests...
2022-09-27T16:01:48.4432010Z ----------------------------------------------------------------------
2022-09-27T16:01:48.4914542Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:49.2604039Z ok (8.462s)
2022-09-27T16:01:49.2604248Z 
2022-09-27T16:01:49.2604661Z ----------------------------------------------------------------------
2022-09-27T16:01:49.2604986Z Ran 1 test in 8.463s
2022-09-27T16:01:49.2605152Z 
2022-09-27T16:01:49.2605246Z OK
2022-09-27T16:01:49.2605381Z 
2022-09-27T16:01:49.2605522Z Generating XML reports...
2022-09-27T16:01:49.2641329Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160140.xml
2022-09-27T16:01:49.8538967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmmz1o4j
2022-09-27T16:01:49.8539849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmmz1o4j/_remote_module_non_scriptable.py
2022-09-27T16:01:50.0057797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxksunls
2022-09-27T16:01:50.0059893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxksunls/_remote_module_non_scriptable.py
2022-09-27T16:01:50.0339925Z [1664294510.033605] [28fd6574525b:32985:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:50.0340432Z [1664294510.033605] [28fd6574525b:32985:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:50.0418578Z [1664294510.041567] [28fd6574525b:32986:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:50.0419078Z [1664294510.041567] [28fd6574525b:32986:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:50.2321890Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:50.2519409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33108
2022-09-27T16:01:50.2526065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33109
2022-09-27T16:01:50.9121136Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:01:50.9219890Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:01:50.9516964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:50.9520103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:51.5327604Z ok (7.210s)
2022-09-27T16:01:51.5327820Z 
2022-09-27T16:01:51.5328195Z ----------------------------------------------------------------------
2022-09-27T16:01:51.5328548Z Ran 1 test in 7.210s
2022-09-27T16:01:51.5329035Z 
2022-09-27T16:01:51.5329146Z OK
2022-09-27T16:01:51.5329286Z 
2022-09-27T16:01:51.5329421Z Generating XML reports...
2022-09-27T16:01:51.5364686Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160144.xml
2022-09-27T16:01:51.5373907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:51.5374388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:51.5378301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:51.5378777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:51.9315122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:51.9315651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:51.9318146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:51.9318626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:51.9598225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:51.9598705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:51.9601431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:51.9601909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:52.0484316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:01:52.0507799Z 
2022-09-27T16:01:52.0508057Z Running tests...
2022-09-27T16:01:52.0508528Z ----------------------------------------------------------------------
2022-09-27T16:01:52.4964518Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:52.4965175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:52.4965961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:52.4966670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:52.5073951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:52.5178193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrw84vwu
2022-09-27T16:01:52.5181075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrw84vwu/_remote_module_non_scriptable.py
2022-09-27T16:01:52.6025441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:52.6234980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uiyyh0k
2022-09-27T16:01:52.6239759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uiyyh0k/_remote_module_non_scriptable.py
2022-09-27T16:01:52.6901817Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6905953Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6910303Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T16:01:52.6912669Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:01:52.6916068Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T16:01:52.6918627Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:01:52.6923296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:52.6924430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:52.6929630Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6933793Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6945580Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6949662Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6962589Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6966993Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6979367Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6985117Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6995393Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:52.6999675Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:01:53.0593203Z ok (4.616s)
2022-09-27T16:01:53.0593408Z 
2022-09-27T16:01:53.0593800Z ----------------------------------------------------------------------
2022-09-27T16:01:53.0594139Z Ran 1 test in 4.616s
2022-09-27T16:01:53.0594305Z 
2022-09-27T16:01:53.0594384Z OK
2022-09-27T16:01:53.0594522Z 
2022-09-27T16:01:53.0594661Z Generating XML reports...
2022-09-27T16:01:53.0629950Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160148.xml
2022-09-27T16:01:53.6866670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:53.6867262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:53.6868497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:53.6868982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:53.7534315Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:53.7735926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33262
2022-09-27T16:01:53.7740884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33263
2022-09-27T16:01:54.1799608Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:01:54.1823084Z 
2022-09-27T16:01:54.1824264Z Running tests...
2022-09-27T16:01:54.1825260Z ----------------------------------------------------------------------
2022-09-27T16:01:55.2682389Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:55.2683628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:55.2684826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:55.2685741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:55.4647601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:55.4648589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:55.4650590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:55.4651613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:55.5522177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:55.5523088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:55.5524116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:55.5525000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:55.7723736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:01:55.7747068Z 
2022-09-27T16:01:55.7747475Z Running tests...
2022-09-27T16:01:55.7748007Z ----------------------------------------------------------------------
2022-09-27T16:01:55.8207996Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:55.8401912Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33363
2022-09-27T16:01:55.8409283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33364
2022-09-27T16:01:55.9739581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:56.0682377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:56.0683198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:56.0753039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:56.0759627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:56.0762415Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:01:56.1697288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:56.1698142Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:01:57.4724482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplaoummnt
2022-09-27T16:01:57.4725600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplaoummnt/_remote_module_non_scriptable.py
2022-09-27T16:01:57.5063445Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:01:57.5263205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33438
2022-09-27T16:01:57.5268835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33439
2022-09-27T16:01:57.5437652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9_6o30b
2022-09-27T16:01:57.5440289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9_6o30b/_remote_module_non_scriptable.py
2022-09-27T16:01:57.5519900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:57.5520891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:57.5524553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:57.5525582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:57.5881890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:57.5882405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:57.5883909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:57.5884384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:58.1016619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:58.1339911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:58.1340734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:58.1422261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:58.1429594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:01:58.2382674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:59.2228079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:59.2231207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:59.2259145Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-09-27T16:01:59.2263539Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-09-27T16:01:59.2272873Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:01:59.2274976Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:01:59.2275974Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:01:59.2276642Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:01:59.2279605Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:01:59.2281049Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:01:59.2312397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:59.2312874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:59.2316023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:59.2316526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:59.2507283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:01:59.2507772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:01:59.2509579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:01:59.2510061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:01:59.2565842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:59.2566367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:01:59.5332426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1p9a41s
2022-09-27T16:01:59.5333564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1p9a41s/_remote_module_non_scriptable.py
2022-09-27T16:01:59.6138879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9j53044
2022-09-27T16:01:59.6139481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9j53044/_remote_module_non_scriptable.py
2022-09-27T16:01:59.6420328Z [1664294519.641547] [28fd6574525b:33364:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:59.6420863Z [1664294519.641547] [28fd6574525b:33364:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:59.6421347Z [1664294519.641703] [28fd6574525b:33363:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:01:59.6421820Z [1664294519.641703] [28fd6574525b:33363:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:01:59.8251583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:01:59.8252401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:01:59.8253199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:59.8253903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:01:59.8260408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:01:59.8261079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:00.0877935Z ok (8.037s)
2022-09-27T16:02:00.0878158Z 
2022-09-27T16:02:00.0878563Z ----------------------------------------------------------------------
2022-09-27T16:02:00.0878885Z Ran 1 test in 8.037s
2022-09-27T16:02:00.0879050Z 
2022-09-27T16:02:00.0879145Z OK
2022-09-27T16:02:00.0879285Z 
2022-09-27T16:02:00.0879431Z Generating XML reports...
2022-09-27T16:02:00.0915375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160152.xml
2022-09-27T16:02:00.1233937Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:02:00.1234837Z   warnings.warn(
2022-09-27T16:02:00.1235927Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:02:00.1236681Z   warnings.warn(
2022-09-27T16:02:01.1522506Z ok (6.970s)
2022-09-27T16:02:01.1522748Z 
2022-09-27T16:02:01.1523138Z ----------------------------------------------------------------------
2022-09-27T16:02:01.1523489Z Ran 1 test in 6.970s
2022-09-27T16:02:01.1523660Z 
2022-09-27T16:02:01.1523753Z OK
2022-09-27T16:02:01.1523898Z 
2022-09-27T16:02:01.1524017Z Generating XML reports...
2022-09-27T16:02:01.1559805Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160154.xml
2022-09-27T16:02:01.3230918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz29gq65v
2022-09-27T16:02:01.3233229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz29gq65v/_remote_module_non_scriptable.py
2022-09-27T16:02:01.4076146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwp59g6qi
2022-09-27T16:02:01.4077516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwp59g6qi/_remote_module_non_scriptable.py
2022-09-27T16:02:01.9285279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:01.9285839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:02.3810374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:02.3811789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:02.3813362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:02.3815147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:02.5381766Z ok (6.763s)
2022-09-27T16:02:02.5382004Z 
2022-09-27T16:02:02.5382579Z ----------------------------------------------------------------------
2022-09-27T16:02:02.5383314Z Ran 1 test in 6.763s
2022-09-27T16:02:02.5383484Z 
2022-09-27T16:02:02.5383587Z OK
2022-09-27T16:02:02.5383726Z 
2022-09-27T16:02:02.5383873Z Generating XML reports...
2022-09-27T16:02:02.5417933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160155.xml
2022-09-27T16:02:02.9532331Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:02.9555034Z 
2022-09-27T16:02:02.9555534Z Running tests...
2022-09-27T16:02:02.9556022Z ----------------------------------------------------------------------
2022-09-27T16:02:02.9576401Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:02.9577207Z 
2022-09-27T16:02:02.9577612Z ----------------------------------------------------------------------
2022-09-27T16:02:02.9577930Z Ran 1 test in 0.002s
2022-09-27T16:02:02.9578104Z 
2022-09-27T16:02:02.9578216Z OK (skipped=1)
2022-09-27T16:02:02.9578375Z 
2022-09-27T16:02:02.9578499Z Generating XML reports...
2022-09-27T16:02:02.9612986Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160202.xml
2022-09-27T16:02:03.3772877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:03.3773815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:03.3774956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:03.3775864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:03.9176524Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:03.9198153Z 
2022-09-27T16:02:03.9198640Z Running tests...
2022-09-27T16:02:03.9199177Z ----------------------------------------------------------------------
2022-09-27T16:02:03.9218775Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:02:03.9219135Z 
2022-09-27T16:02:03.9219414Z ----------------------------------------------------------------------
2022-09-27T16:02:03.9219733Z Ran 1 test in 0.002s
2022-09-27T16:02:03.9219902Z 
2022-09-27T16:02:03.9220018Z OK (skipped=1)
2022-09-27T16:02:03.9220177Z 
2022-09-27T16:02:03.9220305Z Generating XML reports...
2022-09-27T16:02:03.9254207Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160203.xml
2022-09-27T16:02:04.6874065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:04.6874707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:04.6876132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:04.6876607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:04.9886539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:04.9887207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:04.9888562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:04.9889042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:05.2499293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:05.2522746Z 
2022-09-27T16:02:05.2522977Z Running tests...
2022-09-27T16:02:05.2523419Z ----------------------------------------------------------------------
2022-09-27T16:02:05.4795918Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:05.4818465Z 
2022-09-27T16:02:05.4818721Z Running tests...
2022-09-27T16:02:05.4819156Z ----------------------------------------------------------------------
2022-09-27T16:02:05.4839091Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:05.4839992Z 
2022-09-27T16:02:05.4840887Z ----------------------------------------------------------------------
2022-09-27T16:02:05.4841278Z Ran 1 test in 0.002s
2022-09-27T16:02:05.4841445Z 
2022-09-27T16:02:05.4841571Z OK (skipped=1)
2022-09-27T16:02:05.4841728Z 
2022-09-27T16:02:05.4841855Z Generating XML reports...
2022-09-27T16:02:05.4874821Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160205.xml
2022-09-27T16:02:05.9041717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:05.9042232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:05.9043070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:05.9043552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:06.3953145Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:06.3974885Z 
2022-09-27T16:02:06.3975399Z Running tests...
2022-09-27T16:02:06.3975924Z ----------------------------------------------------------------------
2022-09-27T16:02:06.3995996Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:02:06.3996365Z 
2022-09-27T16:02:06.3996886Z ----------------------------------------------------------------------
2022-09-27T16:02:06.3997221Z Ran 1 test in 0.002s
2022-09-27T16:02:06.3997371Z 
2022-09-27T16:02:06.3997482Z OK (skipped=1)
2022-09-27T16:02:06.3997642Z 
2022-09-27T16:02:06.3997768Z Generating XML reports...
2022-09-27T16:02:06.4031951Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160206.xml
2022-09-27T16:02:06.8361797Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:06.8561320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33747
2022-09-27T16:02:06.8568039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33748
2022-09-27T16:02:07.4079918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:07.4081418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:07.4082070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:07.4082561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:08.0157816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:08.0179290Z 
2022-09-27T16:02:08.0179783Z Running tests...
2022-09-27T16:02:08.0180316Z ----------------------------------------------------------------------
2022-09-27T16:02:08.0201112Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:08.0201744Z 
2022-09-27T16:02:08.0202046Z ----------------------------------------------------------------------
2022-09-27T16:02:08.0202387Z Ran 1 test in 0.002s
2022-09-27T16:02:08.0202550Z 
2022-09-27T16:02:08.0202950Z OK (skipped=1)
2022-09-27T16:02:08.0203111Z 
2022-09-27T16:02:08.0203241Z Generating XML reports...
2022-09-27T16:02:08.0235073Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160208.xml
2022-09-27T16:02:08.4710731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:08.4711722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:08.4712886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:08.4713836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:08.5093677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:08.5094636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:08.5097115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:08.5098086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:08.5520964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:08.5521970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:08.5523155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:08.5524120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:09.0166628Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:09.0212093Z 
2022-09-27T16:02:09.0212344Z Running tests...
2022-09-27T16:02:09.0213240Z ----------------------------------------------------------------------
2022-09-27T16:02:09.0769852Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:09.0770529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:09.0771308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:09.0772008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:09.0878386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:09.1785222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:09.9779999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:09.9781040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:09.9782205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:09.9783131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:10.4271648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbue4lybl
2022-09-27T16:02:10.4272866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbue4lybl/_remote_module_non_scriptable.py
2022-09-27T16:02:10.5049227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnetlhhun
2022-09-27T16:02:10.5050921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnetlhhun/_remote_module_non_scriptable.py
2022-09-27T16:02:10.5078138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:10.5102776Z 
2022-09-27T16:02:10.5103262Z Running tests...
2022-09-27T16:02:10.5104046Z ----------------------------------------------------------------------
2022-09-27T16:02:10.5124475Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:10.5124993Z 
2022-09-27T16:02:10.5125281Z ----------------------------------------------------------------------
2022-09-27T16:02:10.5125601Z Ran 1 test in 0.002s
2022-09-27T16:02:10.5125766Z 
2022-09-27T16:02:10.5125875Z OK (skipped=1)
2022-09-27T16:02:10.5126034Z 
2022-09-27T16:02:10.5126160Z Generating XML reports...
2022-09-27T16:02:10.5160198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160210.xml
2022-09-27T16:02:10.5288986Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-09-27T16:02:10.5290816Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-09-27T16:02:10.5308940Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-09-27T16:02:10.5310713Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-09-27T16:02:10.7067794Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:10.7276763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33921
2022-09-27T16:02:10.7282181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33922
2022-09-27T16:02:11.4682126Z ok (6.215s)
2022-09-27T16:02:11.4682598Z 
2022-09-27T16:02:11.4683422Z ----------------------------------------------------------------------
2022-09-27T16:02:11.4684236Z Ran 1 test in 6.215s
2022-09-27T16:02:11.4684602Z 
2022-09-27T16:02:11.4684822Z OK
2022-09-27T16:02:11.4685141Z 
2022-09-27T16:02:11.4746777Z Generating XML reports...
2022-09-27T16:02:11.4748718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160205.xml
2022-09-27T16:02:12.4567197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:12.4568597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:12.4569846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:12.4570811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:12.4577211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:12.4578136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:12.4581204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:12.4582165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:12.6085755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:12.6086747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:12.6088792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:12.6089715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:12.9767968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:12.9895796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:12.9897311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:12.9970796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:12.9977391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:13.0910146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:13.0994367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:13.1018607Z 
2022-09-27T16:02:13.1019222Z Running tests...
2022-09-27T16:02:13.1019730Z ----------------------------------------------------------------------
2022-09-27T16:02:13.1041348Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:13.1041855Z 
2022-09-27T16:02:13.1042224Z ----------------------------------------------------------------------
2022-09-27T16:02:13.1042559Z Ran 1 test in 0.002s
2022-09-27T16:02:13.1042733Z 
2022-09-27T16:02:13.1042844Z OK (skipped=1)
2022-09-27T16:02:13.1042985Z 
2022-09-27T16:02:13.1043113Z Generating XML reports...
2022-09-27T16:02:13.1077509Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160213.xml
2022-09-27T16:02:13.6365600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:13.6366148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:13.6367239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:13.6367727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:14.1982461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:14.2005098Z 
2022-09-27T16:02:14.2005473Z Running tests...
2022-09-27T16:02:14.2006290Z ----------------------------------------------------------------------
2022-09-27T16:02:14.3953185Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmw79e_z9
2022-09-27T16:02:14.3953967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmw79e_z9/_remote_module_non_scriptable.py
2022-09-27T16:02:14.4775024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkrt0kix
2022-09-27T16:02:14.4775991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkrt0kix/_remote_module_non_scriptable.py
2022-09-27T16:02:14.5052298Z [1664294534.504801] [28fd6574525b:33921:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:14.5052816Z [1664294534.504801] [28fd6574525b:33921:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:14.5112524Z [1664294534.510946] [28fd6574525b:33922:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:14.5113032Z [1664294534.510946] [28fd6574525b:33922:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:15.1250246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:15.1251069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:15.1251732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:15.1252224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:15.4399723Z ok (6.418s)
2022-09-27T16:02:15.4399968Z 
2022-09-27T16:02:15.4400596Z ----------------------------------------------------------------------
2022-09-27T16:02:15.4400943Z Ran 1 test in 6.419s
2022-09-27T16:02:15.4401115Z 
2022-09-27T16:02:15.4401219Z OK
2022-09-27T16:02:15.4401338Z 
2022-09-27T16:02:15.4401471Z Generating XML reports...
2022-09-27T16:02:15.4434779Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160209.xml
2022-09-27T16:02:15.6307551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:15.6330475Z 
2022-09-27T16:02:15.6330950Z Running tests...
2022-09-27T16:02:15.6331510Z ----------------------------------------------------------------------
2022-09-27T16:02:15.6353215Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:15.6353734Z 
2022-09-27T16:02:15.6354026Z ----------------------------------------------------------------------
2022-09-27T16:02:15.6354342Z Ran 1 test in 0.002s
2022-09-27T16:02:15.6354510Z 
2022-09-27T16:02:15.6354623Z OK (skipped=1)
2022-09-27T16:02:15.6354780Z 
2022-09-27T16:02:15.6354905Z Generating XML reports...
2022-09-27T16:02:15.6390033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160215.xml
2022-09-27T16:02:15.8475984Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:15.8675828Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34105
2022-09-27T16:02:15.8681229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34106
2022-09-27T16:02:17.5719375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:17.5720136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:17.5721467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:17.5721999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:17.6029936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:17.6030783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:17.6033439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:17.6033945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:17.6652491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:17.6653024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:17.6654371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:17.6654857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:17.6929767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:17.6930280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:17.6932471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:17.6932961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:18.1226737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:18.1227312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:18.1228094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:18.1228811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:18.1235439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:18.1235962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:18.1574088Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:18.1596774Z 
2022-09-27T16:02:18.1597041Z Running tests...
2022-09-27T16:02:18.1597497Z ----------------------------------------------------------------------
2022-09-27T16:02:18.1914216Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:18.1938072Z 
2022-09-27T16:02:18.1938501Z Running tests...
2022-09-27T16:02:18.1939042Z ----------------------------------------------------------------------
2022-09-27T16:02:18.1963263Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:18.1963792Z 
2022-09-27T16:02:18.1964109Z ----------------------------------------------------------------------
2022-09-27T16:02:18.1964430Z Ran 1 test in 0.002s
2022-09-27T16:02:18.1964598Z 
2022-09-27T16:02:18.1964715Z OK (skipped=1)
2022-09-27T16:02:18.1964872Z 
2022-09-27T16:02:18.1964999Z Generating XML reports...
2022-09-27T16:02:18.1996796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160218.xml
2022-09-27T16:02:19.5002922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5er5z84
2022-09-27T16:02:19.5004027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5er5z84/_remote_module_non_scriptable.py
2022-09-27T16:02:19.5588450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fqi59mr
2022-09-27T16:02:19.5589084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fqi59mr/_remote_module_non_scriptable.py
2022-09-27T16:02:19.8187133Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:19.8396267Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34281
2022-09-27T16:02:19.8400446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34282
2022-09-27T16:02:19.9921774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:19.9922373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:20.0149138Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp3fqi59mr'>
2022-09-27T16:02:20.0150234Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:02:20.0151527Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmps5er5z84'>
2022-09-27T16:02:20.0152506Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:02:20.2082251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:20.2083149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:20.2084245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:20.2085497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:20.5789314Z ok (6.378s)
2022-09-27T16:02:20.5792417Z 
2022-09-27T16:02:20.5794853Z ----------------------------------------------------------------------
2022-09-27T16:02:20.5797219Z Ran 1 test in 6.378s
2022-09-27T16:02:20.5799231Z 
2022-09-27T16:02:20.5801518Z OK
2022-09-27T16:02:20.5803428Z 
2022-09-27T16:02:20.5805239Z Generating XML reports...
2022-09-27T16:02:20.5860988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160214.xml
2022-09-27T16:02:20.7263789Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:20.7288674Z 
2022-09-27T16:02:20.7289202Z Running tests...
2022-09-27T16:02:20.7289754Z ----------------------------------------------------------------------
2022-09-27T16:02:20.7312197Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:20.7313099Z 
2022-09-27T16:02:20.7313402Z ----------------------------------------------------------------------
2022-09-27T16:02:20.7313739Z Ran 1 test in 0.002s
2022-09-27T16:02:20.7314010Z 
2022-09-27T16:02:20.7314211Z OK (skipped=1)
2022-09-27T16:02:20.7314395Z 
2022-09-27T16:02:20.7314524Z Generating XML reports...
2022-09-27T16:02:20.7350094Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160220.xml
2022-09-27T16:02:21.4704953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:21.4716850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:21.4717537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:21.4718030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:21.4718615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:21.4719103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:21.4719708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:21.4720441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:21.9887028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:22.0407361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:22.0408285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:22.0494311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:22.0501000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:22.1420323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:22.6825628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:22.6826160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:22.6827355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:22.6827853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:22.7086645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:22.7087487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:22.7089739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:22.7090235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:23.1982220Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:23.2005062Z 
2022-09-27T16:02:23.2005587Z Running tests...
2022-09-27T16:02:23.2006081Z ----------------------------------------------------------------------
2022-09-27T16:02:23.2290012Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:23.2313935Z 
2022-09-27T16:02:23.2314245Z Running tests...
2022-09-27T16:02:23.2314721Z ----------------------------------------------------------------------
2022-09-27T16:02:23.2336753Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:23.2337262Z 
2022-09-27T16:02:23.2337551Z ----------------------------------------------------------------------
2022-09-27T16:02:23.2337868Z Ran 1 test in 0.002s
2022-09-27T16:02:23.2338034Z 
2022-09-27T16:02:23.2338147Z OK (skipped=1)
2022-09-27T16:02:23.2338305Z 
2022-09-27T16:02:23.2338431Z Generating XML reports...
2022-09-27T16:02:23.2373681Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160223.xml
2022-09-27T16:02:23.3669420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8prd8ak
2022-09-27T16:02:23.3670376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8prd8ak/_remote_module_non_scriptable.py
2022-09-27T16:02:23.4505515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1b5ofji
2022-09-27T16:02:23.4506128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1b5ofji/_remote_module_non_scriptable.py
2022-09-27T16:02:23.4848037Z [1664294543.484310] [28fd6574525b:34281:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:23.4848553Z [1664294543.484310] [28fd6574525b:34281:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:23.4868866Z [1664294543.486565] [28fd6574525b:34282:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:23.4869559Z [1664294543.486565] [28fd6574525b:34282:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:23.8330268Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:02:23.8331118Z   warnings.warn(
2022-09-27T16:02:23.8332178Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:02:23.8332925Z   warnings.warn(
2022-09-27T16:02:24.2655236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:24.2656451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:24.2742488Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:02:24.2744114Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:02:24.2919208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:24.2920725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:24.3022904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:24.3025130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:24.8378768Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:24.8523090Z ok (6.692s)
2022-09-27T16:02:24.8523305Z 
2022-09-27T16:02:24.8523731Z ----------------------------------------------------------------------
2022-09-27T16:02:24.8524077Z Ran 1 test in 6.693s
2022-09-27T16:02:24.8524253Z 
2022-09-27T16:02:24.8524331Z OK
2022-09-27T16:02:24.8524479Z 
2022-09-27T16:02:24.8524618Z Generating XML reports...
2022-09-27T16:02:24.8560480Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160218.xml
2022-09-27T16:02:24.8576649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34469
2022-09-27T16:02:24.8583308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34470
2022-09-27T16:02:25.2441748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:25.2442291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:25.2444548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:25.2445357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:25.8663242Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:25.8686501Z 
2022-09-27T16:02:25.8686914Z Running tests...
2022-09-27T16:02:25.8687442Z ----------------------------------------------------------------------
2022-09-27T16:02:25.8708671Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:25.8709181Z 
2022-09-27T16:02:25.8709473Z ----------------------------------------------------------------------
2022-09-27T16:02:25.8709790Z Ran 1 test in 0.002s
2022-09-27T16:02:25.8709958Z 
2022-09-27T16:02:25.8710067Z OK (skipped=1)
2022-09-27T16:02:25.8710227Z 
2022-09-27T16:02:25.8710353Z Generating XML reports...
2022-09-27T16:02:25.8744310Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160225.xml
2022-09-27T16:02:26.5407544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:26.5408962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:26.5410189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:26.5411142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:26.5548233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:26.5549208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:26.5551728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:26.5552732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:27.0438787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:27.0439801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:27.0441369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:27.0442294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:27.1359191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:27.1360593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:27.1362188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:27.1363741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:27.1467670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:27.2373040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:27.5387137Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:27.5411765Z 
2022-09-27T16:02:27.5412284Z Running tests...
2022-09-27T16:02:27.5412787Z ----------------------------------------------------------------------
2022-09-27T16:02:27.5439403Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T16:02:27.5440084Z 
2022-09-27T16:02:27.5440847Z ----------------------------------------------------------------------
2022-09-27T16:02:27.5441171Z Ran 1 test in 0.003s
2022-09-27T16:02:27.5441650Z 
2022-09-27T16:02:27.5441784Z OK (skipped=1)
2022-09-27T16:02:27.5441947Z 
2022-09-27T16:02:27.5442077Z Generating XML reports...
2022-09-27T16:02:27.5477336Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160227.xml
2022-09-27T16:02:27.8658431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:27.8658944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:27.8661352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:27.8661835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:28.3734498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:28.3757804Z 
2022-09-27T16:02:28.3758137Z Running tests...
2022-09-27T16:02:28.3758579Z ----------------------------------------------------------------------
2022-09-27T16:02:28.3778758Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:28.3779493Z 
2022-09-27T16:02:28.3779781Z ----------------------------------------------------------------------
2022-09-27T16:02:28.3780116Z Ran 1 test in 0.002s
2022-09-27T16:02:28.3780263Z 
2022-09-27T16:02:28.3780373Z OK (skipped=1)
2022-09-27T16:02:28.3780529Z 
2022-09-27T16:02:28.3780656Z Generating XML reports...
2022-09-27T16:02:28.3815224Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160228.xml
2022-09-27T16:02:28.4950465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3eg89pd9
2022-09-27T16:02:28.4951124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3eg89pd9/_remote_module_non_scriptable.py
2022-09-27T16:02:28.5510446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotpvbzqi
2022-09-27T16:02:28.5511626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotpvbzqi/_remote_module_non_scriptable.py
2022-09-27T16:02:29.0372888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:29.0374027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:29.0391299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:29.0392320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:29.5711462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:29.5712342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:29.5713862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:29.5714730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:29.9697848Z ok (6.769s)
2022-09-27T16:02:29.9698081Z 
2022-09-27T16:02:29.9698457Z ----------------------------------------------------------------------
2022-09-27T16:02:29.9698808Z Ran 1 test in 6.769s
2022-09-27T16:02:29.9698980Z 
2022-09-27T16:02:29.9699081Z OK
2022-09-27T16:02:29.9699227Z 
2022-09-27T16:02:29.9699369Z Generating XML reports...
2022-09-27T16:02:29.9734570Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160223.xml
2022-09-27T16:02:30.0751406Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:30.0774369Z 
2022-09-27T16:02:30.0774532Z Running tests...
2022-09-27T16:02:30.0774998Z ----------------------------------------------------------------------
2022-09-27T16:02:30.3625304Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:30.3625955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:30.3627437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:30.3627977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:30.8844454Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:30.8867935Z 
2022-09-27T16:02:30.8868074Z Running tests...
2022-09-27T16:02:30.8869403Z ----------------------------------------------------------------------
2022-09-27T16:02:30.8889787Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:02:30.8890610Z 
2022-09-27T16:02:30.8890921Z ----------------------------------------------------------------------
2022-09-27T16:02:30.8891617Z Ran 1 test in 0.002s
2022-09-27T16:02:30.8891786Z 
2022-09-27T16:02:30.8891903Z OK (skipped=1)
2022-09-27T16:02:30.8892063Z 
2022-09-27T16:02:30.8892195Z Generating XML reports...
2022-09-27T16:02:30.8926668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160230.xml
2022-09-27T16:02:31.6727299Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:31.6925516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34717
2022-09-27T16:02:31.6931214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34718
2022-09-27T16:02:32.0479295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:32.0480613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:32.0481858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:32.0482832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:32.6202083Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:32.6224461Z 
2022-09-27T16:02:32.6224588Z Running tests...
2022-09-27T16:02:32.6225632Z ----------------------------------------------------------------------
2022-09-27T16:02:32.8465634Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:32.8466267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:32.8467292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:32.8467789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:33.3475107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:33.3498068Z 
2022-09-27T16:02:33.3498408Z Running tests...
2022-09-27T16:02:33.3498856Z ----------------------------------------------------------------------
2022-09-27T16:02:33.3667109Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:33.3667686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:33.3670900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:33.3671401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:33.4041907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:33.4042474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:33.4044665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:33.4045160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:33.9031819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:33.9564735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:33.9565557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:33.9640396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:33.9648447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:34.0578735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:34.2727797Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:34.2937672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34823
2022-09-27T16:02:34.2943077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34824
2022-09-27T16:02:35.0233525Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:35.0553749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34888
2022-09-27T16:02:35.0558661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34889
2022-09-27T16:02:35.3806391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fqvm5z3
2022-09-27T16:02:35.3807543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fqvm5z3/_remote_module_non_scriptable.py
2022-09-27T16:02:35.4744725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpef75qesf
2022-09-27T16:02:35.4745922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpef75qesf/_remote_module_non_scriptable.py
2022-09-27T16:02:35.5011962Z [1664294555.500741] [28fd6574525b:34717:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:35.5012945Z [1664294555.500741] [28fd6574525b:34717:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:35.5074704Z [1664294555.507122] [28fd6574525b:34718:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:35.5075684Z [1664294555.507122] [28fd6574525b:34718:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:36.0775558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:36.0776073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:36.0776681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:36.0777184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:36.0942088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:36.0942614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:36.0944619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:36.0945118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:36.3946647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:36.3947484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:36.4029493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:36.4032519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:36.6410418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:36.6410966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:36.6411766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:36.6412478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:36.6519396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:36.7289677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:36.7290199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:36.7291864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:36.7292351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:36.7423107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:36.7632076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:02:36.7632591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:02:36.7633374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:02:36.7634091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:02:36.8008334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:36.8008837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:36.8010167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:36.8010656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:37.0046804Z ok (6.927s)
2022-09-27T16:02:37.0047023Z 
2022-09-27T16:02:37.0047379Z ----------------------------------------------------------------------
2022-09-27T16:02:37.0047721Z Ran 1 test in 6.927s
2022-09-27T16:02:37.0047888Z 
2022-09-27T16:02:37.0047989Z OK
2022-09-27T16:02:37.0048126Z 
2022-09-27T16:02:37.0048263Z Generating XML reports...
2022-09-27T16:02:37.0083196Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160230.xml
2022-09-27T16:02:37.2510969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:37.3507939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:37.3508790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:37.3526965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:37.3534260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:37.4521890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:38.2445352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzp_jndmn
2022-09-27T16:02:38.2447443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzp_jndmn/_remote_module_non_scriptable.py
2022-09-27T16:02:38.2700111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g4p2itr
2022-09-27T16:02:38.2701292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g4p2itr/_remote_module_non_scriptable.py
2022-09-27T16:02:38.7612285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgc_h0lk1
2022-09-27T16:02:38.7614155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgc_h0lk1/_remote_module_non_scriptable.py
2022-09-27T16:02:38.7618982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:38.7619495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:38.7639682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:38.7640589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:38.8417867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6t9redb
2022-09-27T16:02:38.8419052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6t9redb/_remote_module_non_scriptable.py
2022-09-27T16:02:39.2223459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:39.2224157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:39.2225222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:39.2225716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:39.6069805Z ok (6.984s)
2022-09-27T16:02:39.6070043Z 
2022-09-27T16:02:39.6070702Z ----------------------------------------------------------------------
2022-09-27T16:02:39.6071045Z Ran 1 test in 6.984s
2022-09-27T16:02:39.6071212Z 
2022-09-27T16:02:39.6071314Z OK
2022-09-27T16:02:39.6071451Z 
2022-09-27T16:02:39.6071588Z Generating XML reports...
2022-09-27T16:02:39.6112497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160232.xml
2022-09-27T16:02:39.7460290Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:39.7483554Z 
2022-09-27T16:02:39.7483951Z Running tests...
2022-09-27T16:02:39.7484424Z ----------------------------------------------------------------------
2022-09-27T16:02:40.5338566Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.5339803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.5478226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.5479240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.5600630Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:02:40.5604140Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:02:40.5940288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.5941347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.6081184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.6082750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.6179284Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:02:40.6180791Z   warnings.warn(
2022-09-27T16:02:40.6183327Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:02:40.6184759Z   warnings.warn(
2022-09-27T16:02:40.6228233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.6229642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.6370641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:40.6371486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:41.2701005Z ok (7.920s)
2022-09-27T16:02:41.2701414Z 
2022-09-27T16:02:41.2702075Z ----------------------------------------------------------------------
2022-09-27T16:02:41.2702741Z Ran 1 test in 7.920s
2022-09-27T16:02:41.2703056Z 
2022-09-27T16:02:41.2703236Z OK
2022-09-27T16:02:41.2703488Z 
2022-09-27T16:02:41.2703748Z Generating XML reports...
2022-09-27T16:02:41.2759499Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160233.xml
2022-09-27T16:02:41.4777644Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:41.4977850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35076
2022-09-27T16:02:41.4983083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35077
2022-09-27T16:02:41.7749118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:41.7749618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:41.7751895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:41.7752411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:42.3748483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:42.3771785Z 
2022-09-27T16:02:42.3772107Z Running tests...
2022-09-27T16:02:42.3772712Z ----------------------------------------------------------------------
2022-09-27T16:02:43.1649382Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:43.1649989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:43.1653001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:43.1653541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:43.2328889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:43.2329383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:43.2330950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:43.2331449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:43.4855384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:43.4855902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:43.4857530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:43.4858222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:43.6751487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:43.7567701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:43.7569060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:43.7663886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:43.7720298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:43.8581005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:43.9953113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:43.9976100Z 
2022-09-27T16:02:43.9976606Z Running tests...
2022-09-27T16:02:43.9977557Z ----------------------------------------------------------------------
2022-09-27T16:02:44.0459184Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:44.0660300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35183
2022-09-27T16:02:44.0666283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35184
2022-09-27T16:02:45.1671171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ifsm1qr
2022-09-27T16:02:45.1672312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ifsm1qr/_remote_module_non_scriptable.py
2022-09-27T16:02:45.2901641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjy6em8db
2022-09-27T16:02:45.2902860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjy6em8db/_remote_module_non_scriptable.py
2022-09-27T16:02:45.3156582Z [1664294565.315171] [28fd6574525b:35076:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:45.3157472Z [1664294565.315171] [28fd6574525b:35076:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:45.3248331Z [1664294565.324373] [28fd6574525b:35077:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:45.3249283Z [1664294565.324373] [28fd6574525b:35077:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:45.6397392Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:45.6601431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35259
2022-09-27T16:02:45.6607586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35260
2022-09-27T16:02:45.7125527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:45.7126059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:45.7127419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:45.7127909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:45.8228342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:45.8228857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:45.8229852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:45.8230335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:45.9606644Z STAGE:2022-09-27 16:02:45 35076:35076 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:02:45 35077:35077 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:45.9607077Z 
2022-09-27T16:02:45.9685110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:45.9685931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:45.9861206Z STAGE:2022-09-27 16:02:45 35077:35077 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:45.9861798Z STAGE:2022-09-27 16:02:45 35076:35076 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:46.0500372Z STAGE:2022-09-27 16:02:46 35076:35076 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:02:46.4428101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:46.4429117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:46.4429929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:46.4430651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:46.4436898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:46.4437417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:46.4438470Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:02:46.4439798Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:02:47.0118792Z STAGE:2022-09-27 16:02:47 35076:35076 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:02:47.3618899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:47.3619452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:47.3620754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:47.3621249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:47.3638561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:47.3639037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:47.3642948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:47.3643442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:47.7120209Z ok (7.963s)
2022-09-27T16:02:47.7120795Z 
2022-09-27T16:02:47.7121458Z ----------------------------------------------------------------------
2022-09-27T16:02:47.7121830Z Ran 1 test in 7.963s
2022-09-27T16:02:47.7122000Z 
2022-09-27T16:02:47.7122096Z OK
2022-09-27T16:02:47.7122244Z 
2022-09-27T16:02:47.7122362Z Generating XML reports...
2022-09-27T16:02:47.7157331Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160239.xml
2022-09-27T16:02:47.8786550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7urhgt8
2022-09-27T16:02:47.8787451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7urhgt8/_remote_module_non_scriptable.py
2022-09-27T16:02:47.8934028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuync6mj
2022-09-27T16:02:47.8935163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuync6mj/_remote_module_non_scriptable.py
2022-09-27T16:02:47.9091648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:47.9092798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:47.9093592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:47.9094257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:47.9101436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:47.9101934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:48.4218054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:48.4219059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:48.4236121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:48.4236808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:48.4503218Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:02:48.4504230Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:02:48.7187776Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:02:48.7188417Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:02:48.7268661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:48.7269599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:48.7286719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:48.7287431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:48.7547701Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:02:48.7548795Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:02:48.9966190Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:02:48.9988973Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:02:49.0081996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:49.0082554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:49.0101550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:49.0102063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:49.3428987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv60bvunv
2022-09-27T16:02:49.3430006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv60bvunv/_remote_module_non_scriptable.py
2022-09-27T16:02:49.3541112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xoanvmg
2022-09-27T16:02:49.3542591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xoanvmg/_remote_module_non_scriptable.py
2022-09-27T16:02:49.8806958Z ok (7.503s)
2022-09-27T16:02:49.8807190Z 
2022-09-27T16:02:49.8807573Z ----------------------------------------------------------------------
2022-09-27T16:02:49.8807920Z Ran 1 test in 7.503s
2022-09-27T16:02:49.8808087Z 
2022-09-27T16:02:49.8808184Z OK
2022-09-27T16:02:49.8808334Z 
2022-09-27T16:02:49.8808474Z Generating XML reports...
2022-09-27T16:02:49.8842885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160242.xml
2022-09-27T16:02:49.9406727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:49.9407427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:49.9409157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:49.9409636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:50.4826469Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:50.4850716Z 
2022-09-27T16:02:50.4851142Z Running tests...
2022-09-27T16:02:50.4851635Z ----------------------------------------------------------------------
2022-09-27T16:02:50.8724098Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... ok (6.874s)
2022-09-27T16:02:50.8724427Z 
2022-09-27T16:02:50.8724786Z ----------------------------------------------------------------------
2022-09-27T16:02:50.8725142Z Ran 1 test in 6.875s
2022-09-27T16:02:50.8725313Z 
2022-09-27T16:02:50.8725411Z OK
2022-09-27T16:02:50.8725550Z 
2022-09-27T16:02:50.8725681Z Generating XML reports...
2022-09-27T16:02:50.8760395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160243.xml
2022-09-27T16:02:52.0049402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:52.0050405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:52.0052163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:52.0053148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:52.2147931Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:52.2350549Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35462
2022-09-27T16:02:52.2357410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35463
2022-09-27T16:02:52.5171367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:02:52.5193571Z 
2022-09-27T16:02:52.5193726Z Running tests...
2022-09-27T16:02:52.5194187Z ----------------------------------------------------------------------
2022-09-27T16:02:53.0000576Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:53.0001201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:53.0002760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:53.0003259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:53.4976224Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:02:53.4998977Z 
2022-09-27T16:02:53.4999250Z Running tests...
2022-09-27T16:02:53.4999697Z ----------------------------------------------------------------------
2022-09-27T16:02:53.8647405Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:53.8648025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:53.8649240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:53.8649717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:53.9005732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:53.9006238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:53.9007946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:53.9008420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:54.1226301Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:54.1421612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35532
2022-09-27T16:02:54.1426902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35533
2022-09-27T16:02:54.3775022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:54.4244452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:54.4245979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:54.4281272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:54.4288155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:54.5260509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:55.2050597Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:02:55.2253926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35601
2022-09-27T16:02:55.2258484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35602
2022-09-27T16:02:55.8060099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9cc2hkl
2022-09-27T16:02:55.8061426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9cc2hkl/_remote_module_non_scriptable.py
2022-09-27T16:02:55.8161258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:55.8161763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:55.8165516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:55.8166001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:55.8186059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:55.8186530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:55.8190703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:55.8191180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:55.9335551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuylg28ba
2022-09-27T16:02:55.9338091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuylg28ba/_remote_module_non_scriptable.py
2022-09-27T16:02:55.9855919Z [1664294575.984853] [28fd6574525b:35462:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:55.9857103Z [1664294575.984853] [28fd6574525b:35462:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:55.9862143Z [1664294575.985646] [28fd6574525b:35463:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:02:55.9862993Z [1664294575.985646] [28fd6574525b:35463:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:02:56.4354830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:56.4355407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:56.4356212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:56.4357183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:56.4465577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:56.4467316Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:02:56.5370258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:56.5371134Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:02:56.8992650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:56.8993171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:56.8994101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:56.8994590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:56.9484420Z ok (6.463s)
2022-09-27T16:02:56.9484643Z 
2022-09-27T16:02:56.9485044Z ----------------------------------------------------------------------
2022-09-27T16:02:56.9485387Z Ran 1 test in 6.463s
2022-09-27T16:02:56.9485564Z 
2022-09-27T16:02:56.9485667Z OK
2022-09-27T16:02:56.9485788Z 
2022-09-27T16:02:56.9485922Z Generating XML reports...
2022-09-27T16:02:56.9537609Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160250.xml
2022-09-27T16:02:56.9641643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:56.9642398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:56.9644398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:56.9644893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:57.4485156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:02:57.5217587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:02:57.5218453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:57.5299619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:02:57.5306757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:02:57.6230096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:02:57.9049498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9wenum7
2022-09-27T16:02:57.9050096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9wenum7/_remote_module_non_scriptable.py
2022-09-27T16:02:58.0089555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_6ge930
2022-09-27T16:02:58.0090967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_6ge930/_remote_module_non_scriptable.py
2022-09-27T16:02:58.5197033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:58.5197948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:58.5213903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:58.5214419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:58.5221174Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:02:58.5222092Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:02:58.5254010Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:02:58.5254716Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:02:58.5255815Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:02:58.5256494Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:02:58.5257142Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:02:58.5258025Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:02:58.8693726Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:02:58.8694912Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:02:58.8780189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:58.8780758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:58.8798479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:58.8799274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:02:58.8807472Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:02:58.8808533Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:02:58.8840542Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:02:58.8841404Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:02:58.8842056Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:02:58.8843124Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:02:59.3583923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:02:59.3584439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:02:59.3585334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:02:59.3585819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:02:59.4326725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaj6df9b6
2022-09-27T16:02:59.4327320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaj6df9b6/_remote_module_non_scriptable.py
2022-09-27T16:02:59.4689111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjag144s
2022-09-27T16:02:59.4689912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjag144s/_remote_module_non_scriptable.py
2022-09-27T16:02:59.7671668Z ok (7.248s)
2022-09-27T16:02:59.7672265Z 
2022-09-27T16:02:59.7672904Z ----------------------------------------------------------------------
2022-09-27T16:02:59.7673282Z Ran 1 test in 7.248s
2022-09-27T16:02:59.7673437Z 
2022-09-27T16:02:59.7673549Z OK
2022-09-27T16:02:59.7673698Z 
2022-09-27T16:02:59.7673832Z Generating XML reports...
2022-09-27T16:02:59.7709028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160252.xml
2022-09-27T16:02:59.8989776Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:02:59.9014394Z 
2022-09-27T16:02:59.9014617Z Running tests...
2022-09-27T16:02:59.9015063Z ----------------------------------------------------------------------
2022-09-27T16:03:00.7255950Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:00.7257167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:00.7347136Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpaj6df9b6'>
2022-09-27T16:03:00.7348093Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:03:00.7349248Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpcjag144s'>
2022-09-27T16:03:00.7350095Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:03:01.3399736Z ok (7.840s)
2022-09-27T16:03:01.3400106Z 
2022-09-27T16:03:01.3401029Z ----------------------------------------------------------------------
2022-09-27T16:03:01.3401695Z Ran 1 test in 7.840s
2022-09-27T16:03:01.3402017Z 
2022-09-27T16:03:01.3402201Z OK
2022-09-27T16:03:01.3402435Z 
2022-09-27T16:03:01.3402661Z Generating XML reports...
2022-09-27T16:03:01.3437900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160253.xml
2022-09-27T16:03:01.6877752Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:01.7078980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35779
2022-09-27T16:03:01.7084919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35780
2022-09-27T16:03:01.9441390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:01.9441972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:01.9443300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:01.9443800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:02.6046347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:02.6187285Z 
2022-09-27T16:03:02.6187756Z Running tests...
2022-09-27T16:03:02.6188898Z ----------------------------------------------------------------------
2022-09-27T16:03:03.4218042Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:03.4218998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:03.4219882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:03.4220362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:03.4429206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:03.4429703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:03.4432346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:03.4432843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:03.5694273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:03.5694816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:03.5695870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:03.5696341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:03.9391403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:03.9580906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:03.9581740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:03.9594162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:03.9601334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:04.0595153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:04.0767037Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:03:04.0790911Z 
2022-09-27T16:03:04.0791409Z Running tests...
2022-09-27T16:03:04.0791945Z ----------------------------------------------------------------------
2022-09-27T16:03:04.0849571Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-09-27T16:03:04.0850014Z 
2022-09-27T16:03:04.0850401Z ----------------------------------------------------------------------
2022-09-27T16:03:04.0850754Z Ran 1 test in 0.006s
2022-09-27T16:03:04.0850920Z 
2022-09-27T16:03:04.0851330Z OK (skipped=1)
2022-09-27T16:03:04.0851513Z 
2022-09-27T16:03:04.0851626Z Generating XML reports...
2022-09-27T16:03:04.0887354Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160304.xml
2022-09-27T16:03:04.1750341Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:04.1954658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35885
2022-09-27T16:03:04.1959923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35886
2022-09-27T16:03:05.3991519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprkgs4m15
2022-09-27T16:03:05.3992143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprkgs4m15/_remote_module_non_scriptable.py
2022-09-27T16:03:05.4828853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jl4idog
2022-09-27T16:03:05.4829844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jl4idog/_remote_module_non_scriptable.py
2022-09-27T16:03:05.5087705Z [1664294585.508378] [28fd6574525b:35779:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:03:05.5088507Z [1664294585.508378] [28fd6574525b:35779:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:03:05.5147904Z [1664294585.514465] [28fd6574525b:35780:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:03:05.5148395Z [1664294585.514465] [28fd6574525b:35780:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:03:05.8902676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:05.8903242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:05.8904194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:05.8904693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:05.9110550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:05.9111055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:05.9112654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:05.9113145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:06.1570098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:06.1570968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:06.1571812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:06.1572318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:06.4188668Z ok (6.517s)
2022-09-27T16:03:06.4189010Z 
2022-09-27T16:03:06.4189552Z ----------------------------------------------------------------------
2022-09-27T16:03:06.4189901Z Ran 1 test in 6.517s
2022-09-27T16:03:06.4190069Z 
2022-09-27T16:03:06.4190166Z OK
2022-09-27T16:03:06.4190301Z 
2022-09-27T16:03:06.4190434Z Generating XML reports...
2022-09-27T16:03:06.4224917Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160259.xml
2022-09-27T16:03:06.4681260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:06.4681819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:06.4682605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:06.4683528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:06.4690800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:06.4691526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:06.4693050Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:03:06.4694180Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:03:06.6553759Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:03:06.6576768Z 
2022-09-27T16:03:06.6577180Z Running tests...
2022-09-27T16:03:06.6577670Z ----------------------------------------------------------------------
2022-09-27T16:03:07.8902774Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmib6urq
2022-09-27T16:03:07.8904137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmib6urq/_remote_module_non_scriptable.py
2022-09-27T16:03:07.9460955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp668_ym1i
2022-09-27T16:03:07.9462195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp668_ym1i/_remote_module_non_scriptable.py
2022-09-27T16:03:08.3014087Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:08.3215004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36037
2022-09-27T16:03:08.3220078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36038
2022-09-27T16:03:08.3827215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:08.3827762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:08.3865738Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-09-27T16:03:08.3866377Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-09-27T16:03:08.3880235Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:03:08.3881153Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:03:08.3881998Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:03:08.3882669Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:03:08.3885250Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:03:08.3885921Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:03:08.4173102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:08.4173660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:08.5848654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:08.5849467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:08.5850405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:08.5851144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:09.0081381Z ok (6.389s)
2022-09-27T16:03:09.0087810Z 
2022-09-27T16:03:09.0093577Z ----------------------------------------------------------------------
2022-09-27T16:03:09.0098549Z Ran 1 test in 6.390s
2022-09-27T16:03:09.0103088Z 
2022-09-27T16:03:09.0108277Z OK
2022-09-27T16:03:09.0112295Z 
2022-09-27T16:03:09.0116999Z Generating XML reports...
2022-09-27T16:03:09.0203139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160302.xml
2022-09-27T16:03:09.2083079Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:03:09.2106289Z 
2022-09-27T16:03:09.2106558Z Running tests...
2022-09-27T16:03:09.2106996Z ----------------------------------------------------------------------
2022-09-27T16:03:10.0153317Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:10.0153932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:10.0155780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:10.0156272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:10.0241819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:10.0242317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:10.0246122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:10.0246619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:10.5319448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:10.5955570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:10.5956401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:10.6028299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:10.6035519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:10.6969361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:10.8941555Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:10.9150582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36150
2022-09-27T16:03:10.9155072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36151
2022-09-27T16:03:11.0638970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:11.0639491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:11.0641302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:11.0641797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:11.5719612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:11.5743094Z 
2022-09-27T16:03:11.5744654Z Running tests...
2022-09-27T16:03:12.0044949Z ----------------------------------------------------------------------
2022-09-27T16:03:12.0045725Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkb7ly4k
2022-09-27T16:03:12.0046735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkb7ly4k/_remote_module_non_scriptable.py
2022-09-27T16:03:12.0961004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3rtt1rl
2022-09-27T16:03:12.0961632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3rtt1rl/_remote_module_non_scriptable.py
2022-09-27T16:03:12.5636746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:12.5637701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:12.5638964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:12.5639491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:12.6385419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:12.6386128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:12.6387224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:12.6387723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:13.1056719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:13.1715387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:13.1716247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:13.1767449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:13.1775697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:13.2222274Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:13.2428788Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36229
2022-09-27T16:03:13.2433691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36230
2022-09-27T16:03:13.2728586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:13.6212571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:13.6213137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:14.3364584Z ok (7.678s)
2022-09-27T16:03:14.3364994Z 
2022-09-27T16:03:14.3365646Z ----------------------------------------------------------------------
2022-09-27T16:03:14.3366274Z Ran 1 test in 7.679s
2022-09-27T16:03:14.3366564Z 
2022-09-27T16:03:14.3366759Z OK
2022-09-27T16:03:14.3367012Z 
2022-09-27T16:03:14.3367255Z Generating XML reports...
2022-09-27T16:03:14.3401531Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160306.xml
2022-09-27T16:03:14.6442332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78cmpq1u
2022-09-27T16:03:14.6442944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78cmpq1u/_remote_module_non_scriptable.py
2022-09-27T16:03:14.7649002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ht9ck4z
2022-09-27T16:03:14.7650003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ht9ck4z/_remote_module_non_scriptable.py
2022-09-27T16:03:14.7923579Z [1664294594.791744] [28fd6574525b:36150:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:03:14.7924127Z [1664294594.791744] [28fd6574525b:36150:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:03:14.7931926Z [1664294594.792800] [28fd6574525b:36151:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:03:14.7932421Z [1664294594.792800] [28fd6574525b:36151:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:03:14.9164781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:14.9165289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:14.9167682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:14.9168197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:14.9198596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:14.9199349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:14.9202989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:14.9203477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:15.4652187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:15.4652754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:15.4653544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:15.4654274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:15.4660491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:15.4661016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:15.6246274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:15.6246846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:16.2270350Z ok (7.016s)
2022-09-27T16:03:16.2270580Z 
2022-09-27T16:03:16.2270994Z ----------------------------------------------------------------------
2022-09-27T16:03:16.2271326Z Ran 1 test in 7.016s
2022-09-27T16:03:16.2271502Z 
2022-09-27T16:03:16.2271596Z OK
2022-09-27T16:03:16.2271734Z 
2022-09-27T16:03:16.2271874Z Generating XML reports...
2022-09-27T16:03:16.2306580Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160309.xml
2022-09-27T16:03:16.5277116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:16.5277646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:16.5279755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:16.5280436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:16.9659851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby9iw6ej
2022-09-27T16:03:16.9661769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby9iw6ej/_remote_module_non_scriptable.py
2022-09-27T16:03:16.9979046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6iitq905
2022-09-27T16:03:16.9981405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6iitq905/_remote_module_non_scriptable.py
2022-09-27T16:03:17.0752183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:03:17.0776325Z 
2022-09-27T16:03:17.0776589Z Running tests...
2022-09-27T16:03:17.0777037Z ----------------------------------------------------------------------
2022-09-27T16:03:17.5528042Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:17.5528688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:17.6313851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:17.6314421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:18.2549607Z ok (6.680s)
2022-09-27T16:03:18.2549852Z 
2022-09-27T16:03:18.2550261Z ----------------------------------------------------------------------
2022-09-27T16:03:18.2550604Z Ran 1 test in 6.681s
2022-09-27T16:03:18.2550773Z 
2022-09-27T16:03:18.2550867Z OK
2022-09-27T16:03:18.2550991Z 
2022-09-27T16:03:18.2551127Z Generating XML reports...
2022-09-27T16:03:18.2587306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160311.xml
2022-09-27T16:03:18.3940619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:18.3941147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:18.3943313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:18.3943806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:18.8518861Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:18.8718551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36427
2022-09-27T16:03:18.8725140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36428
2022-09-27T16:03:18.9275618Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:03:18.9299252Z 
2022-09-27T16:03:18.9299731Z Running tests...
2022-09-27T16:03:18.9300235Z ----------------------------------------------------------------------
2022-09-27T16:03:20.4777493Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:20.4778093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:20.4779598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:20.4780074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:20.5885163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:20.5885668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:20.5887217Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:20.5887803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:20.5888289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:20.6053321Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.675s)
2022-09-27T16:03:20.6053901Z 
2022-09-27T16:03:20.6054174Z ----------------------------------------------------------------------
2022-09-27T16:03:20.6054493Z Ran 1 test in 1.675s
2022-09-27T16:03:20.6054950Z 
2022-09-27T16:03:20.6055083Z OK (skipped=1)
2022-09-27T16:03:20.6055249Z 
2022-09-27T16:03:20.6055375Z Generating XML reports...
2022-09-27T16:03:20.6088086Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160318.xml
2022-09-27T16:03:20.6175480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:20.6175958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:20.6178834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:20.6179327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:20.9770237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:20.9792907Z 
2022-09-27T16:03:20.9793029Z Running tests...
2022-09-27T16:03:20.9793502Z ----------------------------------------------------------------------
2022-09-27T16:03:21.1116600Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:21.1474464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:21.1475262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:21.1521568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:21.1529119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:21.2488129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:21.2647897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:03:21.2648873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:03:21.2649664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:03:21.2650368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:03:21.2651344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:03:21.2752938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:03:21.2753700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:03:21.2754394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:03:22.6761469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4iri4m6
2022-09-27T16:03:22.6762136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4iri4m6/_remote_module_non_scriptable.py
2022-09-27T16:03:22.7236455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4grmrga5
2022-09-27T16:03:22.7237362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4grmrga5/_remote_module_non_scriptable.py
2022-09-27T16:03:22.7463028Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:22.7662744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36581
2022-09-27T16:03:22.7668940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36582
2022-09-27T16:03:22.7827206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:22.7827938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:22.7830260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:22.7830783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:23.3215352Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:03:23.3238362Z 
2022-09-27T16:03:23.3238665Z Running tests...
2022-09-27T16:03:23.3239111Z ----------------------------------------------------------------------
2022-09-27T16:03:23.3825710Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... ok (6.305s)
2022-09-27T16:03:23.3826050Z 
2022-09-27T16:03:23.3826425Z ----------------------------------------------------------------------
2022-09-27T16:03:23.3826767Z Ran 1 test in 6.305s
2022-09-27T16:03:23.3826933Z 
2022-09-27T16:03:23.3827011Z OK
2022-09-27T16:03:23.3827147Z 
2022-09-27T16:03:23.3827291Z Generating XML reports...
2022-09-27T16:03:23.3862996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160317.xml
2022-09-27T16:03:24.4554563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:24.4555087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:24.4556213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:24.4556703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:24.4844000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:24.4844487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:24.4847538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:24.4848033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:25.0594932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:25.0595520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:25.0596306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:25.0597013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:25.0597468Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:25.0702702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:25.0761463Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.752s)
2022-09-27T16:03:25.0762041Z 
2022-09-27T16:03:25.0762322Z ----------------------------------------------------------------------
2022-09-27T16:03:25.0762655Z Ran 1 test in 1.752s
2022-09-27T16:03:25.0762819Z 
2022-09-27T16:03:25.0762928Z OK (skipped=1)
2022-09-27T16:03:25.0763068Z 
2022-09-27T16:03:25.0763195Z Generating XML reports...
2022-09-27T16:03:25.0797637Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160323.xml
2022-09-27T16:03:25.1610421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:25.6380498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:25.6381387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:25.6382331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:25.6382830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:26.1506146Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:03:26.1530160Z 
2022-09-27T16:03:26.1530578Z Running tests...
2022-09-27T16:03:26.1531013Z ----------------------------------------------------------------------
2022-09-27T16:03:26.4594618Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0rm9of3
2022-09-27T16:03:26.4595929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0rm9of3/_remote_module_non_scriptable.py
2022-09-27T16:03:26.5489038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rr44623
2022-09-27T16:03:26.5490152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rr44623/_remote_module_non_scriptable.py
2022-09-27T16:03:27.1031787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:27.1033172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:27.1756135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:27.1756773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:27.2642894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:27.2643411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:27.2645854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:27.2646585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:27.8034382Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:03:27.8057610Z 
2022-09-27T16:03:27.8058562Z Running tests...
2022-09-27T16:03:27.8059767Z ----------------------------------------------------------------------
2022-09-27T16:03:27.8090082Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T16:03:27.8090829Z 
2022-09-27T16:03:27.8091352Z ----------------------------------------------------------------------
2022-09-27T16:03:27.8092013Z Ran 1 test in 0.003s
2022-09-27T16:03:27.8092322Z 
2022-09-27T16:03:27.8092521Z OK (skipped=1)
2022-09-27T16:03:27.8092801Z 
2022-09-27T16:03:27.8093043Z Generating XML reports...
2022-09-27T16:03:27.8129061Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160327.xml
2022-09-27T16:03:27.8273293Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:27.8478586Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36763
2022-09-27T16:03:27.8483366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36764
2022-09-27T16:03:27.8784532Z ok (6.899s)
2022-09-27T16:03:27.8784893Z 
2022-09-27T16:03:27.8785521Z ----------------------------------------------------------------------
2022-09-27T16:03:27.8786112Z Ran 1 test in 6.899s
2022-09-27T16:03:27.8786401Z 
2022-09-27T16:03:27.8786539Z OK
2022-09-27T16:03:27.8786778Z 
2022-09-27T16:03:27.8787010Z Generating XML reports...
2022-09-27T16:03:27.8821344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160320.xml
2022-09-27T16:03:29.5768741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:29.5769568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:29.5770600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:29.5771103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:29.5809716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:29.5810187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:29.5813603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:29.5814094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:29.9368743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:29.9369224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:29.9371024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:29.9371510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:30.0608753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:30.0609262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:30.0611546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:30.0612031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:30.0867913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:30.0906817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:30.0907564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:30.0969047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:30.0975948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:30.1921140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:30.2131397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:03:30.2132192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:03:30.2132964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:03:30.2133693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:03:30.2134625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:03:30.2135929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:03:30.2137136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:03:30.2238022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:03:30.4423929Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:03:30.4449031Z 
2022-09-27T16:03:30.4449171Z Running tests...
2022-09-27T16:03:30.4450019Z ----------------------------------------------------------------------
2022-09-27T16:03:30.5558607Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:30.5581064Z 
2022-09-27T16:03:30.5581209Z Running tests...
2022-09-27T16:03:30.5581910Z ----------------------------------------------------------------------
2022-09-27T16:03:31.5964993Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qlyc7qu
2022-09-27T16:03:31.5965862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qlyc7qu/_remote_module_non_scriptable.py
2022-09-27T16:03:31.6182362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rgv142b
2022-09-27T16:03:31.6183389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rgv142b/_remote_module_non_scriptable.py
2022-09-27T16:03:42.6118808Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:42.6323792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36921
2022-09-27T16:03:42.6329643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36922
2022-09-27T16:03:42.6490891Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:42.6689117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36924
2022-09-27T16:03:42.6694289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36925
2022-09-27T16:03:42.7790272Z ok (16.626s)
2022-09-27T16:03:42.7790603Z 
2022-09-27T16:03:42.7791342Z ----------------------------------------------------------------------
2022-09-27T16:03:42.7791984Z Ran 1 test in 16.626s
2022-09-27T16:03:42.7792156Z 
2022-09-27T16:03:42.7792257Z OK
2022-09-27T16:03:42.7792395Z 
2022-09-27T16:03:42.7792509Z Generating XML reports...
2022-09-27T16:03:42.7828033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160326.xml
2022-09-27T16:03:44.3228945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:44.3229474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:44.3230693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:44.3231162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:44.3401106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:44.3401592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:44.3404683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:44.3405151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:44.3458899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:44.3459371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:44.3462950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:44.3463426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:44.3515587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:44.3516048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:44.3519260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:44.3519724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:44.8604150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:44.8624875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:44.8625705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:44.8647186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:44.8647668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:44.8651303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:44.8651791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:44.8705273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:44.8712098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:44.8715065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:03:44.8805617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:44.8806301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:44.8806997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:44.8807699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:44.8814551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:44.8815022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:44.9639812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:44.9642113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:03:44.9642908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:03:44.9731294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:03:45.3711091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:03:45.3734003Z 
2022-09-27T16:03:45.3734250Z Running tests...
2022-09-27T16:03:45.3734688Z ----------------------------------------------------------------------
2022-09-27T16:03:45.5400211Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... ok (15.095s)
2022-09-27T16:03:45.5400848Z 
2022-09-27T16:03:45.5401268Z ----------------------------------------------------------------------
2022-09-27T16:03:45.5401613Z Ran 1 test in 15.095s
2022-09-27T16:03:45.5401780Z 
2022-09-27T16:03:45.5401878Z OK
2022-09-27T16:03:45.5402058Z 
2022-09-27T16:03:45.5402171Z Generating XML reports...
2022-09-27T16:03:45.5546974Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160330.xml
2022-09-27T16:03:46.3060947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe684dqp7
2022-09-27T16:03:46.3062038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe684dqp7/_remote_module_non_scriptable.py
2022-09-27T16:03:46.3100686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpraoodstd
2022-09-27T16:03:46.3103558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpraoodstd/_remote_module_non_scriptable.py
2022-09-27T16:03:46.8511520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:46.8512376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:46.9916606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:46.9917192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:47.0764160Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:47.0968103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37167
2022-09-27T16:03:47.0972878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37168
2022-09-27T16:03:47.5803278Z ok (17.022s)
2022-09-27T16:03:47.5803495Z 
2022-09-27T16:03:47.5803893Z ----------------------------------------------------------------------
2022-09-27T16:03:47.5804238Z Ran 1 test in 17.022s
2022-09-27T16:03:47.5804408Z 
2022-09-27T16:03:47.5804486Z OK
2022-09-27T16:03:47.5804622Z 
2022-09-27T16:03:47.5804755Z Generating XML reports...
2022-09-27T16:03:47.5840401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160330.xml
2022-09-27T16:03:47.9340533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:47.9341547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:47.9343308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:47.9343787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:48.4425099Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:03:48.4447249Z 
2022-09-27T16:03:48.4447689Z Running tests...
2022-09-27T16:03:48.4448202Z ----------------------------------------------------------------------
2022-09-27T16:03:48.7813301Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:48.7813893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:48.7815748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:48.7816239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:48.8544668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:48.8545190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:48.8547467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:48.8547959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:49.2952829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:49.3762920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:49.3763751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:49.3764486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:49.3771560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:49.3772409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:49.6672609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:49.6673129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:49.6674289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:49.6674813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:50.0918076Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:50.1119970Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37277
2022-09-27T16:03:50.1125790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37278
2022-09-27T16:03:50.1824078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:50.1847424Z 
2022-09-27T16:03:50.1847612Z Running tests...
2022-09-27T16:03:50.1848049Z ----------------------------------------------------------------------
2022-09-27T16:03:50.7550505Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7082_bo8
2022-09-27T16:03:50.7552667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7082_bo8/_remote_module_non_scriptable.py
2022-09-27T16:03:50.7752648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpej9g43ni
2022-09-27T16:03:50.7754076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpej9g43ni/_remote_module_non_scriptable.py
2022-09-27T16:03:51.7494713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:51.7495511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:51.7496442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:51.7497193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:51.7641851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:51.7642672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:51.7644367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:51.7645145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:51.8458343Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:51.8667091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37351
2022-09-27T16:03:51.8672233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37352
2022-09-27T16:03:52.2704082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:52.2926869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:52.2927903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:52.3008871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:52.3016722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:52.3019724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:03:52.3940680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:52.3942328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:03:52.3943924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:03:52.4038884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:03:53.0206687Z ok (4.575s)
2022-09-27T16:03:53.0207055Z 
2022-09-27T16:03:53.0208037Z ----------------------------------------------------------------------
2022-09-27T16:03:53.0208693Z Ran 1 test in 4.576s
2022-09-27T16:03:53.0208980Z 
2022-09-27T16:03:53.0209155Z OK
2022-09-27T16:03:53.0209398Z 
2022-09-27T16:03:53.0209619Z Generating XML reports...
2022-09-27T16:03:53.0258850Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160348.xml
2022-09-27T16:03:53.3110304Z ok (7.937s)
2022-09-27T16:03:53.3110520Z 
2022-09-27T16:03:53.3111183Z ----------------------------------------------------------------------
2022-09-27T16:03:53.3111554Z Ran 1 test in 7.937s
2022-09-27T16:03:53.3111728Z 
2022-09-27T16:03:53.3111822Z OK
2022-09-27T16:03:53.3111947Z 
2022-09-27T16:03:53.3112082Z Generating XML reports...
2022-09-27T16:03:53.3147073Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160345.xml
2022-09-27T16:03:53.5159662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:53.5160748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:53.5162326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:53.5162798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:53.5191201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:53.5191678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:53.5195143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:53.5195613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:54.0534312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:54.0534872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:54.0535664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:54.0536387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:54.0542820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:54.0543298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:03:55.2642889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:55.2643907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:55.2645144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:55.2646151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:55.4868937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:55.4869957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:55.4871130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:55.4872087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:55.5277419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17j4whty
2022-09-27T16:03:55.5278544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17j4whty/_remote_module_non_scriptable.py
2022-09-27T16:03:55.5837317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zw2htoy
2022-09-27T16:03:55.5838634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zw2htoy/_remote_module_non_scriptable.py
2022-09-27T16:03:55.7806702Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:03:55.7828791Z 
2022-09-27T16:03:55.7829248Z Running tests...
2022-09-27T16:03:55.7829757Z ----------------------------------------------------------------------
2022-09-27T16:03:56.0026580Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:03:56.0050492Z 
2022-09-27T16:03:56.0050782Z Running tests...
2022-09-27T16:03:56.0051257Z ----------------------------------------------------------------------
2022-09-27T16:03:56.1043223Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:56.1043931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:56.1775554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:56.1776448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:03:56.7780952Z ok (6.593s)
2022-09-27T16:03:56.7781210Z 
2022-09-27T16:03:56.7781604Z ----------------------------------------------------------------------
2022-09-27T16:03:56.7781947Z Ran 1 test in 6.593s
2022-09-27T16:03:56.7782112Z 
2022-09-27T16:03:56.7782207Z OK
2022-09-27T16:03:56.7782333Z 
2022-09-27T16:03:56.7782472Z Generating XML reports...
2022-09-27T16:03:56.7821102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160350.xml
2022-09-27T16:03:57.5067256Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:57.5260059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37539
2022-09-27T16:03:57.5266511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37540
2022-09-27T16:03:57.6932554Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:03:57.7130157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37542
2022-09-27T16:03:57.7136706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37543
2022-09-27T16:03:59.1699618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:59.1700629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:59.1702506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:59.1703446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:59.2444643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:59.2445717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:59.2447005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:59.2448059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:59.3696636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:59.3697629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:59.3698828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:59.3699754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:59.6022462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:59.6023831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:59.6025084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:59.6026036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:59.6400839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:03:59.6401803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:03:59.6404257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:03:59.6405172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:03:59.6661028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:03:59.6684218Z 
2022-09-27T16:03:59.6684704Z Running tests...
2022-09-27T16:03:59.6685217Z ----------------------------------------------------------------------
2022-09-27T16:03:59.7635540Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:03:59.8700651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:03:59.8701470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:59.8749079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:03:59.8757256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:03:59.9714460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:00.1294139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:00.1752641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:00.1753615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:00.1802591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:00.1809882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:00.2850014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:01.2686772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3yu4th_
2022-09-27T16:04:01.2688005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3yu4th_/_remote_module_non_scriptable.py
2022-09-27T16:04:01.3190941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgsop6hy
2022-09-27T16:04:01.3192168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgsop6hy/_remote_module_non_scriptable.py
2022-09-27T16:04:01.3466457Z [1664294641.345977] [28fd6574525b:37539:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:04:01.3467464Z [1664294641.345977] [28fd6574525b:37539:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:04:01.3576957Z [1664294641.357111] [28fd6574525b:37540:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:04:01.3577981Z [1664294641.357111] [28fd6574525b:37540:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:04:01.3850168Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:01.4061664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37722
2022-09-27T16:04:01.4065989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37723
2022-09-27T16:04:01.5557182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpao7a69iw
2022-09-27T16:04:01.5558361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpao7a69iw/_remote_module_non_scriptable.py
2022-09-27T16:04:01.6511544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uq_5yrv
2022-09-27T16:04:01.6512686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uq_5yrv/_remote_module_non_scriptable.py
2022-09-27T16:04:02.2345875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:02.2346907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:02.2411985Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:04:02.2415492Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:04:02.8383170Z ok (7.055s)
2022-09-27T16:04:02.8383410Z 
2022-09-27T16:04:02.8383812Z ----------------------------------------------------------------------
2022-09-27T16:04:02.8384157Z Ran 1 test in 7.055s
2022-09-27T16:04:02.8384334Z 
2022-09-27T16:04:02.8384412Z OK
2022-09-27T16:04:02.8384546Z 
2022-09-27T16:04:02.8384678Z Generating XML reports...
2022-09-27T16:04:02.8419754Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160355.xml
2022-09-27T16:04:03.1071531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:03.1072034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:03.1073315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:03.1073801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:03.1399715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:03.1400436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:03.1402525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:03.1403011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:03.6855540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:03.6856110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:03.6856903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:03.6857615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:03.6864682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:03.6865206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:04.4283897Z ok (8.423s)
2022-09-27T16:04:04.4284133Z 
2022-09-27T16:04:04.4284541Z ----------------------------------------------------------------------
2022-09-27T16:04:04.4284883Z Ran 1 test in 8.423s
2022-09-27T16:04:04.4285052Z 
2022-09-27T16:04:04.4285129Z OK
2022-09-27T16:04:04.4285274Z 
2022-09-27T16:04:04.4285408Z Generating XML reports...
2022-09-27T16:04:04.4320070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160355.xml
2022-09-27T16:04:05.0561957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:05.0562479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:05.0565068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:05.0565574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:05.2103538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf12iizzn
2022-09-27T16:04:05.2104783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf12iizzn/_remote_module_non_scriptable.py
2022-09-27T16:04:05.2290557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfioc50g1
2022-09-27T16:04:05.2291931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfioc50g1/_remote_module_non_scriptable.py
2022-09-27T16:04:05.5866557Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:05.5892561Z 
2022-09-27T16:04:05.5893277Z Running tests...
2022-09-27T16:04:05.5894221Z ----------------------------------------------------------------------
2022-09-27T16:04:05.5915522Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:04:05.5916244Z 
2022-09-27T16:04:05.5916787Z ----------------------------------------------------------------------
2022-09-27T16:04:05.5917395Z Ran 1 test in 0.002s
2022-09-27T16:04:05.5917716Z 
2022-09-27T16:04:05.5917931Z OK (skipped=1)
2022-09-27T16:04:05.5918233Z 
2022-09-27T16:04:05.5918482Z Generating XML reports...
2022-09-27T16:04:05.5956684Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160405.xml
2022-09-27T16:04:05.7830618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:05.7831538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:05.8592092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:05.8592883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:06.4176552Z ok (6.749s)
2022-09-27T16:04:06.4176944Z 
2022-09-27T16:04:06.4177642Z ----------------------------------------------------------------------
2022-09-27T16:04:06.4178292Z Ran 1 test in 6.749s
2022-09-27T16:04:06.4178599Z 
2022-09-27T16:04:06.4178752Z OK
2022-09-27T16:04:06.4179006Z 
2022-09-27T16:04:06.4179252Z Generating XML reports...
2022-09-27T16:04:06.4220762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160359.xml
2022-09-27T16:04:06.6501344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:06.6502461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:06.6504991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:06.6506146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:07.1509115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:04:07.1533108Z 
2022-09-27T16:04:07.1533359Z Running tests...
2022-09-27T16:04:07.1534045Z ----------------------------------------------------------------------
2022-09-27T16:04:07.5238274Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:07.5239672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:07.5241572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:07.5243817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:08.0521807Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:08.0545576Z 
2022-09-27T16:04:08.0545824Z Running tests...
2022-09-27T16:04:08.0546271Z ----------------------------------------------------------------------
2022-09-27T16:04:08.5416930Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:08.5417830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:08.5419315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:08.5419803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:08.8153837Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:08.8357854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37988
2022-09-27T16:04:08.8363812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37989
2022-09-27T16:04:09.0492700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:09.0515572Z 
2022-09-27T16:04:09.0515967Z Running tests...
2022-09-27T16:04:09.0516432Z ----------------------------------------------------------------------
2022-09-27T16:04:09.6779460Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:09.6979996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38054
2022-09-27T16:04:09.6985189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38055
2022-09-27T16:04:10.5473828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:10.5474828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:10.5476072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:10.5477061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:10.6449642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:10.6450625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:10.6452136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:10.6453112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:10.7301080Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:10.7505442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38121
2022-09-27T16:04:10.7511268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38122
2022-09-27T16:04:11.0593647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:11.1671899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:11.1672763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:11.1707917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:11.1714911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:11.2686581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:11.3307577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:11.3308123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:11.3311221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:11.3311698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:11.4133871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:11.4142550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:11.4143928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:11.4144937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:11.8999769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:11.9848117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:11.9848984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:11.9913814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:11.9921907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:12.0860884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:12.4013918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:12.4014852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:12.4016026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:12.4016961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:12.4181612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:12.4182523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:12.4183686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:12.4184621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:12.5055179Z ok (4.451s)
2022-09-27T16:04:12.5055401Z 
2022-09-27T16:04:12.5056010Z ----------------------------------------------------------------------
2022-09-27T16:04:12.5056403Z Ran 1 test in 4.451s
2022-09-27T16:04:12.5056573Z 
2022-09-27T16:04:12.5056670Z OK
2022-09-27T16:04:12.5056811Z 
2022-09-27T16:04:12.5056950Z Generating XML reports...
2022-09-27T16:04:12.5093003Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160408.xml
2022-09-27T16:04:12.5460170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z52vv_r
2022-09-27T16:04:12.5460805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z52vv_r/_remote_module_non_scriptable.py
2022-09-27T16:04:12.6359749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphqezi3wu
2022-09-27T16:04:12.6361159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphqezi3wu/_remote_module_non_scriptable.py
2022-09-27T16:04:12.9817063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:12.9817618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:12.9818432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:12.9819160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:12.9825287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:12.9826345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:14.4059515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu7vd3yw
2022-09-27T16:04:14.4060824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu7vd3yw/_remote_module_non_scriptable.py
2022-09-27T16:04:14.4439237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk9kksg_
2022-09-27T16:04:14.4440715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk9kksg_/_remote_module_non_scriptable.py
2022-09-27T16:04:14.6912086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:14.6913109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:14.6914348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:14.6915316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:15.0501692Z ok (7.896s)
2022-09-27T16:04:15.0501965Z 
2022-09-27T16:04:15.0502371Z ----------------------------------------------------------------------
2022-09-27T16:04:15.0502713Z Ran 1 test in 7.897s
2022-09-27T16:04:15.0502879Z 
2022-09-27T16:04:15.0502973Z OK
2022-09-27T16:04:15.0503111Z 
2022-09-27T16:04:15.0503238Z Generating XML reports...
2022-09-27T16:04:15.0538037Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160407.xml
2022-09-27T16:04:15.1075929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:15.1076675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:15.1803213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:15.1804091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:15.2369180Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:15.2392331Z 
2022-09-27T16:04:15.2392781Z Running tests...
2022-09-27T16:04:15.2393274Z ----------------------------------------------------------------------
2022-09-27T16:04:15.2412220Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:04:15.2412981Z 
2022-09-27T16:04:15.2413631Z ----------------------------------------------------------------------
2022-09-27T16:04:15.2414032Z Ran 1 test in 0.002s
2022-09-27T16:04:15.2414186Z 
2022-09-27T16:04:15.2414308Z OK (skipped=1)
2022-09-27T16:04:15.2414467Z 
2022-09-27T16:04:15.2414594Z Generating XML reports...
2022-09-27T16:04:15.2451110Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160415.xml
2022-09-27T16:04:15.8626634Z ok (6.811s)
2022-09-27T16:04:15.8626847Z 
2022-09-27T16:04:15.8627269Z ----------------------------------------------------------------------
2022-09-27T16:04:15.8627614Z Ran 1 test in 6.811s
2022-09-27T16:04:15.8627780Z 
2022-09-27T16:04:15.8627873Z OK
2022-09-27T16:04:15.8628009Z 
2022-09-27T16:04:15.8628126Z Generating XML reports...
2022-09-27T16:04:15.8667013Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160409.xml
2022-09-27T16:04:17.2474835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:17.2475344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:17.2476776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:17.2477288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:17.2699604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:17.2700398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:17.2704113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:17.2704883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:17.7437966Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:04:17.7460891Z 
2022-09-27T16:04:17.7461297Z Running tests...
2022-09-27T16:04:17.7461916Z ----------------------------------------------------------------------
2022-09-27T16:04:17.7696159Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:17.7721175Z 
2022-09-27T16:04:17.7721458Z Running tests...
2022-09-27T16:04:17.7722206Z ----------------------------------------------------------------------
2022-09-27T16:04:17.7746501Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:04:17.7747143Z 
2022-09-27T16:04:17.7747448Z ----------------------------------------------------------------------
2022-09-27T16:04:17.7747776Z Ran 1 test in 0.003s
2022-09-27T16:04:17.7747958Z 
2022-09-27T16:04:17.7748155Z OK (skipped=1)
2022-09-27T16:04:17.7748452Z 
2022-09-27T16:04:17.7748647Z Generating XML reports...
2022-09-27T16:04:17.7783574Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160417.xml
2022-09-27T16:04:17.9714596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:17.9715126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:17.9716403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:17.9717003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:18.4707162Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:18.4729444Z 
2022-09-27T16:04:18.4729696Z Running tests...
2022-09-27T16:04:18.4730385Z ----------------------------------------------------------------------
2022-09-27T16:04:19.3142886Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:19.3347382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38418
2022-09-27T16:04:19.3352833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38419
2022-09-27T16:04:19.7283135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:19.7283688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:19.7286133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:19.7286617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:20.1278559Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:20.1474371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38484
2022-09-27T16:04:20.1480748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38485
2022-09-27T16:04:20.3381844Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:20.3405711Z 
2022-09-27T16:04:20.3406315Z Running tests...
2022-09-27T16:04:20.3407209Z ----------------------------------------------------------------------
2022-09-27T16:04:20.3427202Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:04:20.3428273Z 
2022-09-27T16:04:20.3428825Z ----------------------------------------------------------------------
2022-09-27T16:04:20.3429453Z Ran 1 test in 0.002s
2022-09-27T16:04:20.3429766Z 
2022-09-27T16:04:20.3429968Z OK (skipped=1)
2022-09-27T16:04:20.3430256Z 
2022-09-27T16:04:20.3430497Z Generating XML reports...
2022-09-27T16:04:20.3465625Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160420.xml
2022-09-27T16:04:21.0364622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:21.0365596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:21.0366823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:21.0367788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:21.1316547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:21.1317584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:21.1318801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:21.1319775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:21.5970586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:21.7080682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:21.7082245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:21.7084161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:21.7091168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:21.7240677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:21.7241669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:21.7243347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:21.7244260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:21.8093358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:21.8148172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:21.8149159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:21.8152783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:21.8153763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:22.3110042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:22.3111037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:22.3112189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:22.3113118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:22.3685720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:22.3686433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:22.3687769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:22.3689424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:22.3695460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:22.3696328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:22.8278907Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:22.8303557Z 
2022-09-27T16:04:22.8304048Z Running tests...
2022-09-27T16:04:22.8304902Z ----------------------------------------------------------------------
2022-09-27T16:04:22.8324126Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:04:22.8324791Z 
2022-09-27T16:04:22.8325366Z ----------------------------------------------------------------------
2022-09-27T16:04:22.8326024Z Ran 1 test in 0.002s
2022-09-27T16:04:22.8326316Z 
2022-09-27T16:04:22.8326515Z OK (skipped=1)
2022-09-27T16:04:22.8326802Z 
2022-09-27T16:04:22.8327043Z Generating XML reports...
2022-09-27T16:04:22.8362130Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160422.xml
2022-09-27T16:04:23.0521691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd588w2j
2022-09-27T16:04:23.0522920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd588w2j/_remote_module_non_scriptable.py
2022-09-27T16:04:23.1852222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaq30i5vd
2022-09-27T16:04:23.1853458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaq30i5vd/_remote_module_non_scriptable.py
2022-09-27T16:04:23.7860747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfkn1u5_
2022-09-27T16:04:23.7861841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfkn1u5_/_remote_module_non_scriptable.py
2022-09-27T16:04:23.7888197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4iiyh5p
2022-09-27T16:04:23.7891233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4iiyh5p/_remote_module_non_scriptable.py
2022-09-27T16:04:24.3824394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:24.3825118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:24.4572612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:24.4573237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:24.7323511Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:04:24.7417955Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:04:24.7494109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:24.7494910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:24.8900041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:24.8900554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:24.8901963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:24.8902436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:25.1602322Z ok (6.687s)
2022-09-27T16:04:25.1602691Z 
2022-09-27T16:04:25.1603316Z ----------------------------------------------------------------------
2022-09-27T16:04:25.1603914Z Ran 1 test in 6.687s
2022-09-27T16:04:25.1604214Z 
2022-09-27T16:04:25.1604378Z OK
2022-09-27T16:04:25.1604619Z 
2022-09-27T16:04:25.1604850Z Generating XML reports...
2022-09-27T16:04:25.1639589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160418.xml
2022-09-27T16:04:25.3493795Z ok (7.603s)
2022-09-27T16:04:25.3494003Z 
2022-09-27T16:04:25.3494399Z ----------------------------------------------------------------------
2022-09-27T16:04:25.3494741Z Ran 1 test in 7.603s
2022-09-27T16:04:25.3494910Z 
2022-09-27T16:04:25.3495008Z OK
2022-09-27T16:04:25.3495148Z 
2022-09-27T16:04:25.3495267Z Generating XML reports...
2022-09-27T16:04:25.3529645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160417.xml
2022-09-27T16:04:25.4368145Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:25.4391082Z 
2022-09-27T16:04:25.4391246Z Running tests...
2022-09-27T16:04:25.4392313Z ----------------------------------------------------------------------
2022-09-27T16:04:25.4411453Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:04:25.4411790Z 
2022-09-27T16:04:25.4412078Z ----------------------------------------------------------------------
2022-09-27T16:04:25.4412423Z Ran 1 test in 0.002s
2022-09-27T16:04:25.4412572Z 
2022-09-27T16:04:25.4412686Z OK (skipped=1)
2022-09-27T16:04:25.4412845Z 
2022-09-27T16:04:25.4412971Z Generating XML reports...
2022-09-27T16:04:25.4448104Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160425.xml
2022-09-27T16:04:27.3169737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:27.3170266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:27.3171156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:27.3171667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:27.5080607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:27.5081102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:27.5083600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:27.5084096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:27.5411209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:27.5411681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:27.5417904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:27.5418400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:27.8019596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:27.8041857Z 
2022-09-27T16:04:27.8042390Z Running tests...
2022-09-27T16:04:27.8042912Z ----------------------------------------------------------------------
2022-09-27T16:04:28.0024791Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:28.0048017Z 
2022-09-27T16:04:28.0048289Z Running tests...
2022-09-27T16:04:28.0048732Z ----------------------------------------------------------------------
2022-09-27T16:04:28.0066910Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:04:28.0067746Z 
2022-09-27T16:04:28.0068036Z ----------------------------------------------------------------------
2022-09-27T16:04:28.0068401Z Ran 1 test in 0.002s
2022-09-27T16:04:28.0068577Z 
2022-09-27T16:04:28.0068686Z OK (skipped=1)
2022-09-27T16:04:28.0068845Z 
2022-09-27T16:04:28.0068970Z Generating XML reports...
2022-09-27T16:04:28.0104289Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160427.xml
2022-09-27T16:04:28.0686745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:04:28.0710564Z 
2022-09-27T16:04:28.0710890Z Running tests...
2022-09-27T16:04:28.0711334Z ----------------------------------------------------------------------
2022-09-27T16:04:29.3814990Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:29.4015861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38816
2022-09-27T16:04:29.4021292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38817
2022-09-27T16:04:29.6274013Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:29.6472595Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38819
2022-09-27T16:04:29.6478969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38820
2022-09-27T16:04:29.9826126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:29.9826846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:29.9828369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:29.9828866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:30.5961880Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:30.5984468Z 
2022-09-27T16:04:30.5984968Z Running tests...
2022-09-27T16:04:30.5985485Z ----------------------------------------------------------------------
2022-09-27T16:04:30.6003850Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:04:30.6004191Z 
2022-09-27T16:04:30.6004445Z ----------------------------------------------------------------------
2022-09-27T16:04:30.6004790Z Ran 1 test in 0.002s
2022-09-27T16:04:30.6004961Z 
2022-09-27T16:04:30.6005078Z OK (skipped=1)
2022-09-27T16:04:30.6005237Z 
2022-09-27T16:04:30.6005364Z Generating XML reports...
2022-09-27T16:04:30.6039890Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160430.xml
2022-09-27T16:04:31.0921446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:31.0922453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:31.0923608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:31.0924972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:31.1051811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:31.1052729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:31.1056401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:31.1057385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:31.3347831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:31.3348845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:31.3350013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:31.3350969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:31.3989441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:31.3990434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:31.3991583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:31.3992530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:31.6713096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:31.6713689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:31.6714466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:31.6715173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:31.6820917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:31.7727166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:31.8930790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:31.9190723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:31.9191589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:31.9236429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:31.9242815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:32.0204849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:32.5785850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:32.5786360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:32.5788064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:32.5788546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:33.0875890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83k_f5rm
2022-09-27T16:04:33.0876775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83k_f5rm/_remote_module_non_scriptable.py
2022-09-27T16:04:33.1235611Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:33.1260727Z 
2022-09-27T16:04:33.1261264Z Running tests...
2022-09-27T16:04:33.1261741Z ----------------------------------------------------------------------
2022-09-27T16:04:33.1562201Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvaf7c4cd
2022-09-27T16:04:33.1563172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvaf7c4cd/_remote_module_non_scriptable.py
2022-09-27T16:04:33.2681865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy0620zr
2022-09-27T16:04:33.2682470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy0620zr/_remote_module_non_scriptable.py
2022-09-27T16:04:33.3534080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1i7tizo1
2022-09-27T16:04:33.3534700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1i7tizo1/_remote_module_non_scriptable.py
2022-09-27T16:04:33.7209158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:33.7209752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:33.7943278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:33.7943831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:34.5135658Z ok (6.709s)
2022-09-27T16:04:34.5135883Z 
2022-09-27T16:04:34.5136276Z ----------------------------------------------------------------------
2022-09-27T16:04:34.5136626Z Ran 1 test in 6.709s
2022-09-27T16:04:34.5136802Z 
2022-09-27T16:04:34.5136899Z OK
2022-09-27T16:04:34.5137037Z 
2022-09-27T16:04:34.5137179Z Generating XML reports...
2022-09-27T16:04:34.5172882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160427.xml
2022-09-27T16:04:34.6196015Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:04:34.6196850Z   warnings.warn(
2022-09-27T16:04:34.6197916Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:04:34.6198959Z   warnings.warn(
2022-09-27T16:04:34.8905329Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:34.9113209Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39045
2022-09-27T16:04:34.9118031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39046
2022-09-27T16:04:35.7627922Z ok (7.691s)
2022-09-27T16:04:35.7630595Z 
2022-09-27T16:04:35.7633344Z ----------------------------------------------------------------------
2022-09-27T16:04:35.7636051Z Ran 1 test in 7.691s
2022-09-27T16:04:35.7639344Z 
2022-09-27T16:04:35.7643337Z OK
2022-09-27T16:04:35.7646626Z 
2022-09-27T16:04:35.7649233Z Generating XML reports...
2022-09-27T16:04:35.7694387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160428.xml
2022-09-27T16:04:36.6230835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:36.6231362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:36.6233520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:36.6234359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:36.6695386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:36.6695870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:36.6698655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:36.6699149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:36.7814600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:36.7815081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:36.7817569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:36.7818066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:37.1866905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:37.2225048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:37.2226087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:37.2271877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:37.2279227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:37.2284051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:04:37.2779019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:37.2804788Z 
2022-09-27T16:04:37.2805214Z Running tests...
2022-09-27T16:04:37.2806085Z ----------------------------------------------------------------------
2022-09-27T16:04:37.3237179Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:37.3239246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:04:37.3240637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:04:37.3297156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:04:37.8189092Z ok (4.692s)
2022-09-27T16:04:37.8189337Z 
2022-09-27T16:04:37.8189742Z ----------------------------------------------------------------------
2022-09-27T16:04:37.8190102Z Ran 1 test in 4.693s
2022-09-27T16:04:37.8190269Z 
2022-09-27T16:04:37.8190369Z OK
2022-09-27T16:04:37.8190507Z 
2022-09-27T16:04:37.8190626Z Generating XML reports...
2022-09-27T16:04:37.8224663Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160433.xml
2022-09-27T16:04:37.9441729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:37.9442228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:37.9444588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:37.9445092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:38.4557574Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:04:38.4579822Z 
2022-09-27T16:04:38.4580080Z Running tests...
2022-09-27T16:04:38.4580801Z ----------------------------------------------------------------------
2022-09-27T16:04:38.8680759Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:38.8964663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39211
2022-09-27T16:04:38.9051454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39219
2022-09-27T16:04:39.9907474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:39.9908321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:39.9910060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:39.9910562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:40.0902454Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:40.1102959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39284
2022-09-27T16:04:40.1109347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39285
2022-09-27T16:04:40.4943456Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:40.4967388Z 
2022-09-27T16:04:40.4967732Z Running tests...
2022-09-27T16:04:40.4968370Z ----------------------------------------------------------------------
2022-09-27T16:04:40.4992845Z   test_get_future (__main__.TestDistBackendWithSpawn) ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T16:04:40.4993168Z 
2022-09-27T16:04:40.4993458Z ----------------------------------------------------------------------
2022-09-27T16:04:40.4993787Z Ran 1 test in 0.003s
2022-09-27T16:04:40.4993975Z 
2022-09-27T16:04:40.4994091Z OK (skipped=1)
2022-09-27T16:04:40.4994253Z 
2022-09-27T16:04:40.4994381Z Generating XML reports...
2022-09-27T16:04:40.5029995Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160440.xml
2022-09-27T16:04:40.5734626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:40.5735147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:40.5736487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:40.5736978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:40.5988517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:40.5989322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:40.5990666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:40.5991168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:41.1869137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:41.1870238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:41.1871669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:41.1873096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:41.1976880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:41.2882198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:41.8028133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:41.8029560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:41.8030780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:41.8031748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:41.8166136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:41.8167135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:41.8168669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:41.8169631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:42.3321862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:42.3459897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:42.3461316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:42.3524905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:42.3531549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:42.4476000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:42.5469755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:42.5470790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:42.5471988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:42.5472952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:42.6302408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_wnohnk
2022-09-27T16:04:42.6303570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_wnohnk/_remote_module_non_scriptable.py
2022-09-27T16:04:42.6981238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxezsu_c
2022-09-27T16:04:42.6981881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxezsu_c/_remote_module_non_scriptable.py
2022-09-27T16:04:43.0892027Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:43.0916253Z 
2022-09-27T16:04:43.0916411Z Running tests...
2022-09-27T16:04:43.0917372Z ----------------------------------------------------------------------
2022-09-27T16:04:43.2672476Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:43.2673114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:43.3433473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:43.3434048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:43.7252904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ju6dijh
2022-09-27T16:04:43.7253506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ju6dijh/_remote_module_non_scriptable.py
2022-09-27T16:04:43.8311399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfcdijmw
2022-09-27T16:04:43.8312671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfcdijmw/_remote_module_non_scriptable.py
2022-09-27T16:04:44.0165988Z ok (6.736s)
2022-09-27T16:04:44.0166210Z 
2022-09-27T16:04:44.0166603Z ----------------------------------------------------------------------
2022-09-27T16:04:44.0167229Z Ran 1 test in 6.736s
2022-09-27T16:04:44.0167386Z 
2022-09-27T16:04:44.0167483Z OK
2022-09-27T16:04:44.0167624Z 
2022-09-27T16:04:44.0167758Z Generating XML reports...
2022-09-27T16:04:44.0204153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160437.xml
2022-09-27T16:04:44.2502321Z STAGE:2022-09-27 16:04:44 39285:39285 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:44.2831392Z STAGE:2022-09-27 16:04:44 39284:39284 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:44.8186014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:44.8186588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:44.8254459Z STAGE:2022-09-27 16:04:44 39285:39285 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:44.8255073Z STAGE:2022-09-27 16:04:44 39284:39284 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:44.8445406Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:44.8494873Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:04:44.8496447Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:04:44.8530639Z STAGE:2022-09-27 16:04:44 39285:39285 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:44.8531239Z STAGE:2022-09-27 16:04:44 39284:39284 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:44.8559843Z STAGE:2022-09-27 16:04:44 39285:39285 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:44.8570901Z STAGE:2022-09-27 16:04:44 39284:39284 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:44.8645862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39455
2022-09-27T16:04:44.8652913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39456
2022-09-27T16:04:45.5228081Z ok (7.065s)
2022-09-27T16:04:45.5228372Z 
2022-09-27T16:04:45.5228970Z ----------------------------------------------------------------------
2022-09-27T16:04:45.5229324Z Ran 1 test in 7.065s
2022-09-27T16:04:45.5229493Z 
2022-09-27T16:04:45.5229589Z OK
2022-09-27T16:04:45.5229723Z 
2022-09-27T16:04:45.5229859Z Generating XML reports...
2022-09-27T16:04:45.5265991Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160438.xml
2022-09-27T16:04:46.2886062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:46.2886561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:46.2888634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:46.2889135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:46.5619420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:46.5620429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:46.5621368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:46.5621857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:46.6088657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:46.6089455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:46.6091136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:46.6091625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:46.8382814Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:46.8404628Z 
2022-09-27T16:04:46.8405185Z Running tests...
2022-09-27T16:04:46.8405817Z ----------------------------------------------------------------------
2022-09-27T16:04:47.1164447Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:47.1240808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:47.1241854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:47.1266422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:47.1273971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:47.2255764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:47.7373553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:47.7374059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:47.7374939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:47.7375431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:47.8725669Z ok (4.780s)
2022-09-27T16:04:47.8726019Z 
2022-09-27T16:04:47.8726815Z ----------------------------------------------------------------------
2022-09-27T16:04:47.8727615Z Ran 1 test in 4.781s
2022-09-27T16:04:47.8727806Z 
2022-09-27T16:04:47.8727882Z OK
2022-09-27T16:04:47.8728019Z 
2022-09-27T16:04:47.8728156Z Generating XML reports...
2022-09-27T16:04:47.8761431Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160443.xml
2022-09-27T16:04:48.2543226Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:04:48.2566723Z 
2022-09-27T16:04:48.2567332Z Running tests...
2022-09-27T16:04:48.2567811Z ----------------------------------------------------------------------
2022-09-27T16:04:48.3934543Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:48.4130306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39592
2022-09-27T16:04:48.4135550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39593
2022-09-27T16:04:49.9036960Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:49.9234922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39688
2022-09-27T16:04:49.9240678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39689
2022-09-27T16:04:49.9996679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:49.9997183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:49.9999399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:49.9999992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:50.0434973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:50.0435464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:50.0437673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:50.0438158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:50.0498049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:50.0498523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:50.0501744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:50.0502225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:50.4868156Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:50.4892124Z 
2022-09-27T16:04:50.4892463Z Running tests...
2022-09-27T16:04:50.4893165Z ----------------------------------------------------------------------
2022-09-27T16:04:50.6074251Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:50.6091690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:50.6095686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:50.6198500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:50.6269229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:50.7113843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:51.5932706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:51.5934107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:51.5935405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:51.5936360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:51.6467358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:51.6468332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:51.6469459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:51.6470404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:52.0609627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl76hpvid
2022-09-27T16:04:52.0610855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl76hpvid/_remote_module_non_scriptable.py
2022-09-27T16:04:52.0985085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprf_87s4n
2022-09-27T16:04:52.0986293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprf_87s4n/_remote_module_non_scriptable.py
2022-09-27T16:04:52.1471480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:52.1718093Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:52.1838992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:52.1840866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:52.1876586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:52.1884379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:52.1916946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39805
2022-09-27T16:04:52.1924509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39806
2022-09-27T16:04:52.2855293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:52.6827154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:52.6827716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:52.7596011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:52.7596556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:53.5249805Z ok (6.684s)
2022-09-27T16:04:53.5250026Z 
2022-09-27T16:04:53.5250435Z ----------------------------------------------------------------------
2022-09-27T16:04:53.5250783Z Ran 1 test in 6.684s
2022-09-27T16:04:53.5250972Z 
2022-09-27T16:04:53.5251051Z OK
2022-09-27T16:04:53.5251187Z 
2022-09-27T16:04:53.5251325Z Generating XML reports...
2022-09-27T16:04:53.5287820Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160446.xml
2022-09-27T16:04:53.7484905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt45sgdls
2022-09-27T16:04:53.7486131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt45sgdls/_remote_module_non_scriptable.py
2022-09-27T16:04:53.7652161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2lgrcgi
2022-09-27T16:04:53.7654497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2lgrcgi/_remote_module_non_scriptable.py
2022-09-27T16:04:53.9427717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:53.9428525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:53.9429854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:53.9430356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:53.9535148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:53.9535639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:53.9537975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:53.9538466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:54.2038807Z STAGE:2022-09-27 16:04:54 39689:39689 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:54.2098959Z STAGE:2022-09-27 16:04:54 39688:39688 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:54.4817501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:04:54.4818549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:04:54.4820318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:54.4821702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:04:54.4825201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:04:54.4826163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:04:54.4827772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:04:54.4828783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:04:54.4830124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:04:54.4831533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:04:54.7804414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:54.7804948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:04:54.7883405Z STAGE:2022-09-27 16:04:54 39689:39689 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:54.7885209Z STAGE:2022-09-27 16:04:54 39688:39688 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:54.8168487Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:04:54.8170118Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:04:54.8207529Z STAGE:2022-09-27 16:04:54 39688:39688 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:54.8208169Z STAGE:2022-09-27 16:04:54 39689:39689 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:04:54.8250509Z STAGE:2022-09-27 16:04:54 39689:39689 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:54.8253502Z STAGE:2022-09-27 16:04:54 39688:39688 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:04:54.8994656Z ok (4.410s)
2022-09-27T16:04:54.8994873Z 
2022-09-27T16:04:54.8995440Z ----------------------------------------------------------------------
2022-09-27T16:04:54.8995770Z Ran 1 test in 4.410s
2022-09-27T16:04:54.8995939Z 
2022-09-27T16:04:54.8996041Z OK
2022-09-27T16:04:54.8996178Z 
2022-09-27T16:04:54.8996313Z Generating XML reports...
2022-09-27T16:04:54.9048035Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160450.xml
2022-09-27T16:04:55.4361728Z ok (7.179s)
2022-09-27T16:04:55.4362109Z 
2022-09-27T16:04:55.4362550Z ----------------------------------------------------------------------
2022-09-27T16:04:55.4362896Z Ran 1 test in 7.179s
2022-09-27T16:04:55.4363048Z 
2022-09-27T16:04:55.4363143Z OK
2022-09-27T16:04:55.4363536Z 
2022-09-27T16:04:55.4363678Z Generating XML reports...
2022-09-27T16:04:55.4398678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160448.xml
2022-09-27T16:04:55.6907419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:55.6907919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:55.6910122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:55.6910628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:56.2453636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:04:56.2476465Z 
2022-09-27T16:04:56.2476954Z Running tests...
2022-09-27T16:04:56.2477474Z ----------------------------------------------------------------------
2022-09-27T16:04:57.0957732Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:57.0958380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:57.0959420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:57.0959947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:57.5258209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:57.5258728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:57.5261312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:57.5261814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:57.5909182Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:04:57.5932503Z 
2022-09-27T16:04:57.5932922Z Running tests...
2022-09-27T16:04:57.5933451Z ----------------------------------------------------------------------
2022-09-27T16:04:57.8693479Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:57.8896130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39996
2022-09-27T16:04:57.8901781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39997
2022-09-27T16:04:58.0249123Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:04:58.0271199Z 
2022-09-27T16:04:58.0271368Z Running tests...
2022-09-27T16:04:58.0272078Z ----------------------------------------------------------------------
2022-09-27T16:04:59.2403380Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:59.2606957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40062
2022-09-27T16:04:59.2612131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40063
2022-09-27T16:04:59.5511577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:59.5512130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:59.5513409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:59.5513938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:59.5683669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:04:59.5684162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:04:59.5686414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:04:59.5686904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:04:59.6751156Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:04:59.6952422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40067
2022-09-27T16:04:59.6957779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40068
2022-09-27T16:05:00.1720561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:00.1721220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:00.1722062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:00.1722778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:00.1728601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:00.1731131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:00.9149760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:00.9150732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:00.9151952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:00.9152901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:01.0475875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:01.0476716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:01.0477942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:01.0478887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:01.3959554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:01.3960883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:01.3962098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:01.3963009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:01.4345370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:01.4346362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:01.4347596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:01.4348538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:01.4483387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:01.5733280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:01.5734916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:01.5782388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jrovnfd
2022-09-27T16:05:01.5784638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jrovnfd/_remote_module_non_scriptable.py
2022-09-27T16:05:01.5805877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:01.5812453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:01.5816662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:05:01.6121535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdvvz6ic
2022-09-27T16:05:01.6122724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdvvz6ic/_remote_module_non_scriptable.py
2022-09-27T16:05:01.6746460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:01.6748122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:05:01.6749617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:05:01.6833560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:05:01.9442264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:01.9934600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:01.9936060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:01.9950628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:01.9958321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:02.0950119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:02.1500666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:02.1501605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:02.1690804Z ok (4.575s)
2022-09-27T16:05:02.1691031Z 
2022-09-27T16:05:02.1691431Z ----------------------------------------------------------------------
2022-09-27T16:05:02.1691773Z Ran 1 test in 4.576s
2022-09-27T16:05:02.1691943Z 
2022-09-27T16:05:02.1692042Z OK
2022-09-27T16:05:02.1692295Z 
2022-09-27T16:05:02.1692488Z Generating XML reports...
2022-09-27T16:05:02.1887330Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160457.xml
2022-09-27T16:05:02.2231467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:02.2232612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:02.9023171Z ok (6.654s)
2022-09-27T16:05:02.9023544Z 
2022-09-27T16:05:02.9024054Z ----------------------------------------------------------------------
2022-09-27T16:05:02.9024413Z Ran 1 test in 6.654s
2022-09-27T16:05:02.9024589Z 
2022-09-27T16:05:02.9024686Z OK
2022-09-27T16:05:02.9024820Z 
2022-09-27T16:05:02.9024957Z Generating XML reports...
2022-09-27T16:05:02.9060556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160456.xml
2022-09-27T16:05:03.5229062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pal2_08
2022-09-27T16:05:03.5230020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pal2_08/_remote_module_non_scriptable.py
2022-09-27T16:05:03.5469259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa97hdmeq
2022-09-27T16:05:03.5470469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa97hdmeq/_remote_module_non_scriptable.py
2022-09-27T16:05:04.4078788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:04.4079790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:04.4080929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:04.4081654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:04.9186316Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:04.9209421Z 
2022-09-27T16:05:04.9209696Z Running tests...
2022-09-27T16:05:04.9210155Z ----------------------------------------------------------------------
2022-09-27T16:05:04.9238187Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T16:05:04.9238538Z 
2022-09-27T16:05:04.9238824Z ----------------------------------------------------------------------
2022-09-27T16:05:04.9239144Z Ran 1 test in 0.003s
2022-09-27T16:05:04.9239310Z 
2022-09-27T16:05:04.9239420Z OK (skipped=1)
2022-09-27T16:05:04.9239585Z 
2022-09-27T16:05:04.9239713Z Generating XML reports...
2022-09-27T16:05:04.9274590Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160504.xml
2022-09-27T16:05:05.0164447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:05.0164950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:05.0166670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:05.0167145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:05.2079580Z ok (7.181s)
2022-09-27T16:05:05.2079802Z 
2022-09-27T16:05:05.2080697Z ----------------------------------------------------------------------
2022-09-27T16:05:05.2081256Z Ran 1 test in 7.181s
2022-09-27T16:05:05.2081414Z 
2022-09-27T16:05:05.2081513Z OK
2022-09-27T16:05:05.2081668Z 
2022-09-27T16:05:05.2081809Z Generating XML reports...
2022-09-27T16:05:05.2116615Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160458.xml
2022-09-27T16:05:05.5173267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:05:05.5195202Z 
2022-09-27T16:05:05.5195453Z Running tests...
2022-09-27T16:05:05.5195900Z ----------------------------------------------------------------------
2022-09-27T16:05:06.8234643Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:06.8235633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:06.8236548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:06.8237054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:07.1383820Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:07.1576055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40392
2022-09-27T16:05:07.1582126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40393
2022-09-27T16:05:07.2948310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:07.2948827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:07.2951217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:07.2951729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:07.3254123Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:07.3278480Z 
2022-09-27T16:05:07.3278832Z Running tests...
2022-09-27T16:05:07.3279270Z ----------------------------------------------------------------------
2022-09-27T16:05:07.7983726Z   test_irecv (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:05:07.8025843Z 
2022-09-27T16:05:07.8029751Z Running tests...
2022-09-27T16:05:07.8030808Z ----------------------------------------------------------------------
2022-09-27T16:05:08.7915645Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:08.7916339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:08.7918492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:08.7918995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:08.8204877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:08.8205379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:08.8207963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:08.8208458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:09.0061639Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:09.0264221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40462
2022-09-27T16:05:09.0269401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40463
2022-09-27T16:05:09.3654975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:09.3655537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:09.3656341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:09.3657028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:09.3663741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:09.3664570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:09.4591731Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:09.4790969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40475
2022-09-27T16:05:09.4798074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40476
2022-09-27T16:05:10.7917411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwa7s2o4d
2022-09-27T16:05:10.7918595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwa7s2o4d/_remote_module_non_scriptable.py
2022-09-27T16:05:10.8054848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:10.8055817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:10.8057851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:10.8058817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:10.8675206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:10.8675771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:10.8677316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:10.8677809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:10.8880577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdj6jkfe
2022-09-27T16:05:10.8881606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdj6jkfe/_remote_module_non_scriptable.py
2022-09-27T16:05:11.2139617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:11.2140624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:11.2141808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:11.2142686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:11.3247747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:11.3636658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:11.3637587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:11.3638991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:11.3639870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:11.3799104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:11.3799898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:11.3857593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:11.3858438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:11.3859146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:11.3865935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:11.3866431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:11.4494112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:11.4494921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:11.7501959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:11.9069020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:11.9070775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:11.9128528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:11.9135089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:12.0082530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:12.1700805Z ok (6.650s)
2022-09-27T16:05:12.1701012Z 
2022-09-27T16:05:12.1701422Z ----------------------------------------------------------------------
2022-09-27T16:05:12.1701768Z Ran 1 test in 6.650s
2022-09-27T16:05:12.1701938Z 
2022-09-27T16:05:12.1702038Z OK
2022-09-27T16:05:12.1702175Z 
2022-09-27T16:05:12.1704334Z Generating XML reports...
2022-09-27T16:05:12.1739114Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160505.xml
2022-09-27T16:05:12.9284987Z [1664294712.928012] [28fd6574525b:40463:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:05:12.9285531Z [1664294712.928012] [28fd6574525b:40463:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:05:12.9286337Z [1664294712.928318] [28fd6574525b:40462:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:05:12.9286798Z [1664294712.928318] [28fd6574525b:40462:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:05:13.3628822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqz_0rb1
2022-09-27T16:05:13.3629435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqz_0rb1/_remote_module_non_scriptable.py
2022-09-27T16:05:13.4818339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsi49rp08
2022-09-27T16:05:13.4820506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsi49rp08/_remote_module_non_scriptable.py
2022-09-27T16:05:13.5776712Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:05:13.5778361Z   warnings.warn(
2022-09-27T16:05:13.5780474Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:05:13.5781915Z   warnings.warn(
2022-09-27T16:05:13.8388153Z ok (6.511s)
2022-09-27T16:05:13.8388374Z 
2022-09-27T16:05:13.8388764Z ----------------------------------------------------------------------
2022-09-27T16:05:13.8389126Z Ran 1 test in 6.511s
2022-09-27T16:05:13.8389277Z 
2022-09-27T16:05:13.8389380Z OK
2022-09-27T16:05:13.8389519Z 
2022-09-27T16:05:13.8389657Z Generating XML reports...
2022-09-27T16:05:13.8444676Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160507.xml
2022-09-27T16:05:14.0498124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:14.0498694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:14.0553953Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:05:14.0555591Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:05:14.0664705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:14.0665232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:14.0729982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:14.0731351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:14.4108766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:14.4109277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:14.4112017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:14.4112535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:14.6922537Z ok (6.889s)
2022-09-27T16:05:14.6922767Z 
2022-09-27T16:05:14.6923147Z ----------------------------------------------------------------------
2022-09-27T16:05:14.6923490Z Ran 1 test in 6.890s
2022-09-27T16:05:14.6923662Z 
2022-09-27T16:05:14.6923759Z OK
2022-09-27T16:05:14.6923895Z 
2022-09-27T16:05:14.6924032Z Generating XML reports...
2022-09-27T16:05:14.6959090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160507.xml
2022-09-27T16:05:14.9455552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:05:14.9479513Z 
2022-09-27T16:05:14.9479658Z Running tests...
2022-09-27T16:05:14.9480767Z ----------------------------------------------------------------------
2022-09-27T16:05:15.9802753Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:15.9803828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:15.9804889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:15.9805386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:16.4723186Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:16.4746092Z 
2022-09-27T16:05:16.4746221Z Running tests...
2022-09-27T16:05:16.4746964Z ----------------------------------------------------------------------
2022-09-27T16:05:16.6680492Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:16.6885632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40768
2022-09-27T16:05:16.6891529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40769
2022-09-27T16:05:16.7296073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:16.7296570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:16.7298705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:16.7299225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:17.2291910Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:05:17.2314888Z 
2022-09-27T16:05:17.2315173Z Running tests...
2022-09-27T16:05:17.2315611Z ----------------------------------------------------------------------
2022-09-27T16:05:18.1475553Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:18.1675518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40835
2022-09-27T16:05:18.1681676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40836
2022-09-27T16:05:18.3177287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:18.3177810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:18.3179311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:18.3180063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:18.3541541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:18.3542027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:18.3544541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:18.3545011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:18.9093242Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:18.9336007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40903
2022-09-27T16:05:18.9341319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40904
2022-09-27T16:05:18.9496042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:18.9496615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:18.9497412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:18.9498138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:18.9606111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:19.0516151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:19.8992087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:19.8992975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:19.8994058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:19.8994919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:19.9792537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:19.9793495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:19.9794697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:19.9795654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:20.3481113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkctome06
2022-09-27T16:05:20.3482561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkctome06/_remote_module_non_scriptable.py
2022-09-27T16:05:20.4302220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkchryiob
2022-09-27T16:05:20.4303398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkchryiob/_remote_module_non_scriptable.py
2022-09-27T16:05:20.4432259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:20.5201914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:20.5203423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:20.5245306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:20.5252474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:20.6215458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:20.6533291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:20.6534644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:20.6536000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:20.6536674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:20.6537446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:20.6537975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:20.6541136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:20.6541905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:20.9151312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:20.9152387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:20.9316296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:20.9317344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:20.9454636Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:05:20.9457686Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:05:20.9843403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:20.9844471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:21.0009415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:21.0010843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:21.0120109Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:05:21.0122074Z   warnings.warn(
2022-09-27T16:05:21.0124298Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:05:21.0125805Z   warnings.warn(
2022-09-27T16:05:21.0179816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:21.0180849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:21.0347174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:21.0348211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:21.2475828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:21.2537397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:21.2538222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:21.2578990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:21.2586260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:21.3554238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:21.6004744Z ok (6.652s)
2022-09-27T16:05:21.6004967Z 
2022-09-27T16:05:21.6005342Z ----------------------------------------------------------------------
2022-09-27T16:05:21.6005691Z Ran 1 test in 6.652s
2022-09-27T16:05:21.6005862Z 
2022-09-27T16:05:21.6005957Z OK
2022-09-27T16:05:21.6006094Z 
2022-09-27T16:05:21.6006233Z Generating XML reports...
2022-09-27T16:05:21.6041385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160514.xml
2022-09-27T16:05:22.1102866Z [1664294722.109689] [28fd6574525b:40835:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:05:22.1103933Z [1664294722.109689] [28fd6574525b:40835:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:05:22.1141178Z [1664294722.113755] [28fd6574525b:40836:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:05:22.1142025Z [1664294722.113755] [28fd6574525b:40836:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:05:22.6679213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmit3i2a
2022-09-27T16:05:22.6679819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmit3i2a/_remote_module_non_scriptable.py
2022-09-27T16:05:22.8153436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2tlvuaz
2022-09-27T16:05:22.8154836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2tlvuaz/_remote_module_non_scriptable.py
2022-09-27T16:05:22.9622988Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:05:22.9623858Z   warnings.warn(
2022-09-27T16:05:22.9624934Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:05:22.9625676Z   warnings.warn(
2022-09-27T16:05:23.0792859Z ok (6.604s)
2022-09-27T16:05:23.0793103Z 
2022-09-27T16:05:23.0793502Z ----------------------------------------------------------------------
2022-09-27T16:05:23.0793827Z Ran 1 test in 6.605s
2022-09-27T16:05:23.0793999Z 
2022-09-27T16:05:23.0794112Z OK
2022-09-27T16:05:23.0794256Z 
2022-09-27T16:05:23.0794398Z Generating XML reports...
2022-09-27T16:05:23.0830533Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160516.xml
2022-09-27T16:05:23.4178227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:23.4178781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:23.7849965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:23.7850859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:23.7852009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:23.7852906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:23.9458677Z ok (6.714s)
2022-09-27T16:05:23.9459051Z 
2022-09-27T16:05:23.9459705Z ----------------------------------------------------------------------
2022-09-27T16:05:23.9460335Z Ran 1 test in 6.714s
2022-09-27T16:05:23.9460639Z 
2022-09-27T16:05:23.9460817Z OK
2022-09-27T16:05:23.9461077Z 
2022-09-27T16:05:23.9461314Z Generating XML reports...
2022-09-27T16:05:23.9495576Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160517.xml
2022-09-27T16:05:24.3539525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:05:24.3560384Z 
2022-09-27T16:05:24.3560681Z Running tests...
2022-09-27T16:05:24.3561120Z ----------------------------------------------------------------------
2022-09-27T16:05:25.3212382Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:25.3213591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:25.3214848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:25.3215848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:25.8144994Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:25.8168119Z 
2022-09-27T16:05:25.8168620Z Running tests...
2022-09-27T16:05:25.8169114Z ----------------------------------------------------------------------
2022-09-27T16:05:25.8990063Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:25.9189907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41113
2022-09-27T16:05:25.9195675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41114
2022-09-27T16:05:26.0523475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:26.0524458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:26.0525904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:26.0526859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:26.5711810Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:05:26.5751324Z 
2022-09-27T16:05:26.5753521Z Running tests...
2022-09-27T16:05:26.5756453Z ----------------------------------------------------------------------
2022-09-27T16:05:27.4476670Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:27.4677179Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41180
2022-09-27T16:05:27.4682468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41181
2022-09-27T16:05:27.5566114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:27.5567012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:27.5568234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:27.5568727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:27.5587505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:27.5587983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:27.5591493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:27.5591990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:28.0930350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:28.0935142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:28.0938649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:28.0941814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:28.0995216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:28.1258212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:28.2399197Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:28.2606152Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41257
2022-09-27T16:05:28.2612450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41258
2022-09-27T16:05:29.1833028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:29.1833575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:29.1835078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:29.1835736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:29.2276555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:29.2277063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:29.2278551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:29.2279328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:29.5320867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t288cl2
2022-09-27T16:05:29.5321489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t288cl2/_remote_module_non_scriptable.py
2022-09-27T16:05:29.5376351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rjxa5ko
2022-09-27T16:05:29.5379176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rjxa5ko/_remote_module_non_scriptable.py
2022-09-27T16:05:29.7203233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:29.7496309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:29.7497158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:29.7507344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:29.7514519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:29.7588816Z STAGE:2022-09-27 16:05:29 41181:41181 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:05:29.8508653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:29.8573467Z STAGE:2022-09-27 16:05:29 41180:41180 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:05:29.9116653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:29.9117157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:29.9119124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:29.9119624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:29.9442653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:29.9443153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:29.9446688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:29.9447164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:30.4784332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:30.5050877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:30.5051755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:30.5090360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:30.5097603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:30.6066085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:31.0321761Z ok (6.676s)
2022-09-27T16:05:31.0321989Z 
2022-09-27T16:05:31.0322386Z ----------------------------------------------------------------------
2022-09-27T16:05:31.0322734Z Ran 1 test in 6.676s
2022-09-27T16:05:31.0322884Z 
2022-09-27T16:05:31.0322988Z OK
2022-09-27T16:05:31.0323128Z 
2022-09-27T16:05:31.0323266Z Generating XML reports...
2022-09-27T16:05:31.0359489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160524.xml
2022-09-27T16:05:31.5832677Z [1664294731.582768] [28fd6574525b:41180:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:05:31.5833504Z [1664294731.582768] [28fd6574525b:41180:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:05:31.6360971Z [1664294731.635597] [28fd6574525b:41181:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:05:31.6361533Z [1664294731.635597] [28fd6574525b:41181:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:05:31.9331227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrps1p9q
2022-09-27T16:05:31.9331848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrps1p9q/_remote_module_non_scriptable.py
2022-09-27T16:05:32.0187598Z STAGE:2022-09-27 16:05:32 41180:41180 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:05:32.0188285Z STAGE:2022-09-27 16:05:32 41181:41181 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:05:32.0261753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3nwgacx
2022-09-27T16:05:32.0264810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3nwgacx/_remote_module_non_scriptable.py
2022-09-27T16:05:32.6073604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:32.6074158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:32.6155850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:32.6158128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:33.1719743Z ok (6.597s)
2022-09-27T16:05:33.1720006Z 
2022-09-27T16:05:33.1720886Z ----------------------------------------------------------------------
2022-09-27T16:05:33.1721415Z Ran 1 test in 6.597s
2022-09-27T16:05:33.1721587Z 
2022-09-27T16:05:33.1721690Z OK
2022-09-27T16:05:33.1721809Z 
2022-09-27T16:05:33.1721947Z Generating XML reports...
2022-09-27T16:05:33.1755443Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160526.xml
2022-09-27T16:05:33.2805462Z ok (7.463s)
2022-09-27T16:05:33.2805660Z 
2022-09-27T16:05:33.2806051Z ----------------------------------------------------------------------
2022-09-27T16:05:33.2806375Z Ran 1 test in 7.464s
2022-09-27T16:05:33.2806550Z 
2022-09-27T16:05:33.2806645Z OK
2022-09-27T16:05:33.2806784Z 
2022-09-27T16:05:33.2806917Z Generating XML reports...
2022-09-27T16:05:33.2812926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:33.2813392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:33.2817180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:33.2817681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:33.2841459Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160525.xml
2022-09-27T16:05:33.8048451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:05:33.8070266Z 
2022-09-27T16:05:33.8070568Z Running tests...
2022-09-27T16:05:33.8071023Z ----------------------------------------------------------------------
2022-09-27T16:05:35.3790465Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:35.3791520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:35.3792126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:35.3792623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:35.4503724Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:35.4704521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41459
2022-09-27T16:05:35.4710031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41460
2022-09-27T16:05:35.4817158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:35.4817636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:35.4821085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:35.4821563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:35.8604380Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:05:35.8626229Z 
2022-09-27T16:05:35.8626490Z Running tests...
2022-09-27T16:05:35.8626957Z ----------------------------------------------------------------------
2022-09-27T16:05:35.9666790Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:35.9689720Z 
2022-09-27T16:05:35.9689934Z Running tests...
2022-09-27T16:05:35.9690384Z ----------------------------------------------------------------------
2022-09-27T16:05:37.1219399Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:37.1220013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:37.1221342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:37.1221842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:37.1402720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:37.1403216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:37.1406512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:37.1407015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:37.4814488Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:37.5013577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41529
2022-09-27T16:05:37.5018954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41530
2022-09-27T16:05:37.5679079Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:37.5875017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41532
2022-09-27T16:05:37.5880483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41533
2022-09-27T16:05:37.6808931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:37.6809494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:37.6810304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:37.6811017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:37.6817812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:37.6818316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:39.2615630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:39.2616599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:39.2618194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:39.2619161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:39.2867640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:39.2868533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:39.2869696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:39.2870605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:39.3264447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:39.3265417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:39.3267305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:39.3268198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:39.3483317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:39.3484312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:39.3486024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:39.3487005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:39.6095924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w260oer
2022-09-27T16:05:39.6096977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w260oer/_remote_module_non_scriptable.py
2022-09-27T16:05:39.6112303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4f18ctv
2022-09-27T16:05:39.6115279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4f18ctv/_remote_module_non_scriptable.py
2022-09-27T16:05:39.6320767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:39.6321605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:39.6422490Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp0w260oer'>
2022-09-27T16:05:39.6423290Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:05:39.6423889Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpe4f18ctv'>
2022-09-27T16:05:39.6424351Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:05:39.8122480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:39.8286291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:39.8287118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:39.8325101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:39.8332034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:39.8736547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:39.8811576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:39.8812386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:39.8838135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:39.8845192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:39.8917540Z STAGE:2022-09-27 16:05:39 41533:41533 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:05:39.9300631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:39.9826338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:39.9899512Z STAGE:2022-09-27 16:05:39 41532:41532 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:05:40.2818646Z ok (6.474s)
2022-09-27T16:05:40.2818856Z 
2022-09-27T16:05:40.2819387Z ----------------------------------------------------------------------
2022-09-27T16:05:40.2819944Z Ran 1 test in 6.475s
2022-09-27T16:05:40.2820114Z 
2022-09-27T16:05:40.2820217Z OK
2022-09-27T16:05:40.2820361Z 
2022-09-27T16:05:40.2820482Z Generating XML reports...
2022-09-27T16:05:40.2857107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160533.xml
2022-09-27T16:05:41.2776191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjakcej8
2022-09-27T16:05:41.2777916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjakcej8/_remote_module_non_scriptable.py
2022-09-27T16:05:41.3335508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0u4ge7s
2022-09-27T16:05:41.3336369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0u4ge7s/_remote_module_non_scriptable.py
2022-09-27T16:05:41.7259387Z STAGE:2022-09-27 16:05:41 41530:41530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:05:41.7265587Z [1664294741.726244] [28fd6574525b:41532:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:05:41.7266109Z [1664294741.726244] [28fd6574525b:41532:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:05:41.7396090Z STAGE:2022-09-27 16:05:41 41529:41529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:05:41.7471412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:41.7471953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:41.7642999Z STAGE:2022-09-27 16:05:41 41529:41529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:05:41.7644717Z STAGE:2022-09-27 16:05:41 41530:41530 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:05:41.8237007Z [1664294741.823199] [28fd6574525b:41533:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:05:41.8237569Z [1664294741.823199] [28fd6574525b:41533:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:05:41.8268096Z STAGE:2022-09-27 16:05:41 41529:41529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:05:42.2477257Z STAGE:2022-09-27 16:05:42 41532:41532 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:05:42.2477917Z STAGE:2022-09-27 16:05:42 41533:41533 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:05:42.5426195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:42.5426947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:42.5427581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:42.5428303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:42.5906239Z STAGE:2022-09-27 16:05:42 41529:41529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:05:43.1098949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:05:43.1129266Z 
2022-09-27T16:05:43.1129800Z Running tests...
2022-09-27T16:05:43.1130429Z ----------------------------------------------------------------------
2022-09-27T16:05:43.2151448Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... ok (7.352s)
2022-09-27T16:05:43.2151786Z 
2022-09-27T16:05:43.2152162Z ----------------------------------------------------------------------
2022-09-27T16:05:43.2152498Z Ran 1 test in 7.352s
2022-09-27T16:05:43.2152671Z 
2022-09-27T16:05:43.2152763Z OK
2022-09-27T16:05:43.2152882Z 
2022-09-27T16:05:43.2153019Z Generating XML reports...
2022-09-27T16:05:43.2189315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160535.xml
2022-09-27T16:05:43.5016368Z ok (7.532s)
2022-09-27T16:05:43.5016778Z 
2022-09-27T16:05:43.5017543Z ----------------------------------------------------------------------
2022-09-27T16:05:43.5017962Z Ran 1 test in 7.532s
2022-09-27T16:05:43.5018133Z 
2022-09-27T16:05:43.5018228Z OK
2022-09-27T16:05:43.5018365Z 
2022-09-27T16:05:43.5018787Z Generating XML reports...
2022-09-27T16:05:43.5054480Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160535.xml
2022-09-27T16:05:44.8304868Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:44.8508155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41813
2022-09-27T16:05:44.8513783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41814
2022-09-27T16:05:45.3071207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:45.3071702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:45.3074088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:45.3074589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:45.6270620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:45.6271162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:45.6272237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:45.6272733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:45.9124294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:05:45.9145917Z 
2022-09-27T16:05:45.9146225Z Running tests...
2022-09-27T16:05:45.9147164Z ----------------------------------------------------------------------
2022-09-27T16:05:46.1185526Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:46.1207220Z 
2022-09-27T16:05:46.1207695Z Running tests...
2022-09-27T16:05:46.1208634Z ----------------------------------------------------------------------
2022-09-27T16:05:46.1226847Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:05:46.1227468Z 
2022-09-27T16:05:46.1227944Z ----------------------------------------------------------------------
2022-09-27T16:05:46.1228435Z Ran 1 test in 0.002s
2022-09-27T16:05:46.1228604Z 
2022-09-27T16:05:46.1228766Z OK (skipped=1)
2022-09-27T16:05:46.1229062Z 
2022-09-27T16:05:46.1229293Z Generating XML reports...
2022-09-27T16:05:46.1263342Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160546.xml
2022-09-27T16:05:46.4814139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:46.4815125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:46.4816415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:46.4817333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:46.5540168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:46.5540995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:46.5542004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:46.5542789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:47.0943800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:47.0945116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:47.0946390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:47.0947387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:47.1134836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:47.1256086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdoy2y3i_
2022-09-27T16:05:47.1259025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdoy2y3i_/_remote_module_non_scriptable.py
2022-09-27T16:05:47.1962256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:47.2072102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hgyy7tp
2022-09-27T16:05:47.2074411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hgyy7tp/_remote_module_non_scriptable.py
2022-09-27T16:05:47.2277500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:47.2278063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:47.5337570Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:47.5542461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41957
2022-09-27T16:05:47.5547083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41958
2022-09-27T16:05:47.7589585Z ok (4.646s)
2022-09-27T16:05:47.7589807Z 
2022-09-27T16:05:47.7590356Z ----------------------------------------------------------------------
2022-09-27T16:05:47.7590843Z Ran 1 test in 4.646s
2022-09-27T16:05:47.7591011Z 
2022-09-27T16:05:47.7591089Z OK
2022-09-27T16:05:47.7591229Z 
2022-09-27T16:05:47.7591363Z Generating XML reports...
2022-09-27T16:05:47.7644478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160543.xml
2022-09-27T16:05:48.0994172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:48.0994742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:48.0997009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:48.0997501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:48.6825662Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:48.6847818Z 
2022-09-27T16:05:48.6848249Z Running tests...
2022-09-27T16:05:48.6848759Z ----------------------------------------------------------------------
2022-09-27T16:05:48.6866588Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:05:48.6867033Z 
2022-09-27T16:05:48.6867919Z ----------------------------------------------------------------------
2022-09-27T16:05:48.6868554Z Ran 1 test in 0.002s
2022-09-27T16:05:48.6868862Z 
2022-09-27T16:05:48.6869070Z OK (skipped=1)
2022-09-27T16:05:48.6869360Z 
2022-09-27T16:05:48.6869586Z Generating XML reports...
2022-09-27T16:05:48.6902594Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160548.xml
2022-09-27T16:05:49.2974130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:49.2974660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:49.2976804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:49.2977321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:49.3229307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:49.3230095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:49.3233957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:49.3234447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:49.8154826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:49.8240127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:49.8241140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:49.8255450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:49.8261914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:49.9254659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:49.9408757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:49.9409255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:49.9411719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:49.9412201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:50.4586687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:05:50.4609242Z 
2022-09-27T16:05:50.4609447Z Running tests...
2022-09-27T16:05:50.4609910Z ----------------------------------------------------------------------
2022-09-27T16:05:50.6148728Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:50.6149341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:50.6151312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:50.6151812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:51.1404386Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:51.1427619Z 
2022-09-27T16:05:51.1427997Z Running tests...
2022-09-27T16:05:51.1450433Z ----------------------------------------------------------------------
2022-09-27T16:05:51.1451395Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:05:51.1451760Z 
2022-09-27T16:05:51.1452033Z ----------------------------------------------------------------------
2022-09-27T16:05:51.1452375Z Ran 1 test in 0.002s
2022-09-27T16:05:51.1452540Z 
2022-09-27T16:05:51.1452632Z OK (skipped=1)
2022-09-27T16:05:51.1452788Z 
2022-09-27T16:05:51.1452914Z Generating XML reports...
2022-09-27T16:05:51.1486902Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160551.xml
2022-09-27T16:05:51.2071266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vlv_oav
2022-09-27T16:05:51.2072436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vlv_oav/_remote_module_non_scriptable.py
2022-09-27T16:05:51.2583890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlxkn0te
2022-09-27T16:05:51.2584740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlxkn0te/_remote_module_non_scriptable.py
2022-09-27T16:05:52.0606160Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:52.0802238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42104
2022-09-27T16:05:52.0809064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42105
2022-09-27T16:05:53.0667384Z ok (7.152s)
2022-09-27T16:05:53.0667775Z 
2022-09-27T16:05:53.0668457Z ----------------------------------------------------------------------
2022-09-27T16:05:53.0669129Z Ran 1 test in 7.152s
2022-09-27T16:05:53.0669431Z 
2022-09-27T16:05:53.0669604Z OK
2022-09-27T16:05:53.0669867Z 
2022-09-27T16:05:53.0670117Z Generating XML reports...
2022-09-27T16:05:53.0704364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160545.xml
2022-09-27T16:05:53.1761670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:53.1762705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:53.1764476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:53.1765442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:53.6809384Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:53.6830691Z 
2022-09-27T16:05:53.6831155Z Running tests...
2022-09-27T16:05:53.6832050Z ----------------------------------------------------------------------
2022-09-27T16:05:53.6856685Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:05:53.6857432Z 
2022-09-27T16:05:53.6857880Z ----------------------------------------------------------------------
2022-09-27T16:05:53.6858223Z Ran 1 test in 0.003s
2022-09-27T16:05:53.6858391Z 
2022-09-27T16:05:53.6858520Z OK (skipped=1)
2022-09-27T16:05:53.6858681Z 
2022-09-27T16:05:53.6858810Z Generating XML reports...
2022-09-27T16:05:53.6891526Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160553.xml
2022-09-27T16:05:53.7712970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:53.7713489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:53.7715346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:53.7715823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:53.7948202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:53.7948720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:53.7951123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:53.7951635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:54.3847540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:54.3848139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:54.3848932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:54.3849642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:54.3955230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:05:54.4860505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:55.2815709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:55.2816413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:55.2818890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:55.2819374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:55.6282651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:55.6283175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:55.6284647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:55.6285121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:55.7458000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyu_37n0
2022-09-27T16:05:55.7458603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyu_37n0/_remote_module_non_scriptable.py
2022-09-27T16:05:55.8032272Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:05:55.8055891Z 
2022-09-27T16:05:55.8056324Z Running tests...
2022-09-27T16:05:55.8056771Z ----------------------------------------------------------------------
2022-09-27T16:05:55.9027449Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9iutu5az
2022-09-27T16:05:55.9028505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9iutu5az/_remote_module_non_scriptable.py
2022-09-27T16:05:56.1435336Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:56.1458856Z 
2022-09-27T16:05:56.1459316Z Running tests...
2022-09-27T16:05:56.1459847Z ----------------------------------------------------------------------
2022-09-27T16:05:56.1483025Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:05:56.1483393Z 
2022-09-27T16:05:56.1483647Z ----------------------------------------------------------------------
2022-09-27T16:05:56.1483991Z Ran 1 test in 0.003s
2022-09-27T16:05:56.1484153Z 
2022-09-27T16:05:56.1484274Z OK (skipped=1)
2022-09-27T16:05:56.1484435Z 
2022-09-27T16:05:56.1484556Z Generating XML reports...
2022-09-27T16:05:56.1527284Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160556.xml
2022-09-27T16:05:56.3527697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:56.3528272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:05:56.9920738Z ok (6.531s)
2022-09-27T16:05:56.9920998Z 
2022-09-27T16:05:56.9921403Z ----------------------------------------------------------------------
2022-09-27T16:05:56.9921742Z Ran 1 test in 6.531s
2022-09-27T16:05:56.9921919Z 
2022-09-27T16:05:56.9922012Z OK
2022-09-27T16:05:56.9922130Z 
2022-09-27T16:05:56.9922263Z Generating XML reports...
2022-09-27T16:05:56.9957334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160550.xml
2022-09-27T16:05:57.5341737Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:05:57.5542109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42322
2022-09-27T16:05:57.5548013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42323
2022-09-27T16:05:58.0965153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:58.0966362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:58.0974805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:58.0976385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:58.6971421Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:05:58.6993476Z 
2022-09-27T16:05:58.6993600Z Running tests...
2022-09-27T16:05:58.6994520Z ----------------------------------------------------------------------
2022-09-27T16:05:58.7014157Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:05:58.7014951Z 
2022-09-27T16:05:58.7015485Z ----------------------------------------------------------------------
2022-09-27T16:05:58.7015870Z Ran 1 test in 0.002s
2022-09-27T16:05:58.7016037Z 
2022-09-27T16:05:58.7016148Z OK (skipped=1)
2022-09-27T16:05:58.7016306Z 
2022-09-27T16:05:58.7016449Z Generating XML reports...
2022-09-27T16:05:58.7050186Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160558.xml
2022-09-27T16:05:59.2384307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:59.2384926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:59.2386503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:59.2387335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:59.2442755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:59.2443549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:59.2447130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:59.2447904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:59.2583227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:05:59.2584074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:05:59.2587321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:05:59.2588099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:05:59.7505676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:05:59.7627046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:05:59.7628566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:59.7707978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:05:59.7713892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:05:59.8471860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:05:59.8494879Z 
2022-09-27T16:05:59.8495376Z Running tests...
2022-09-27T16:05:59.8496306Z ----------------------------------------------------------------------
2022-09-27T16:05:59.8640081Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:00.6619716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:00.6620241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:00.6621798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:00.6622638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:01.0834255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6hjnjbe
2022-09-27T16:06:01.0835295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6hjnjbe/_remote_module_non_scriptable.py
2022-09-27T16:06:01.1807999Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:01.1830367Z 
2022-09-27T16:06:01.1830589Z Running tests...
2022-09-27T16:06:01.1831523Z ----------------------------------------------------------------------
2022-09-27T16:06:01.1852656Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:06:01.1853417Z 
2022-09-27T16:06:01.1853896Z ----------------------------------------------------------------------
2022-09-27T16:06:01.1854239Z Ran 1 test in 0.002s
2022-09-27T16:06:01.1854403Z 
2022-09-27T16:06:01.1854512Z OK (skipped=1)
2022-09-27T16:06:01.1854679Z 
2022-09-27T16:06:01.1854789Z Generating XML reports...
2022-09-27T16:06:01.1889282Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160601.xml
2022-09-27T16:06:01.2034652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhjhl94c
2022-09-27T16:06:01.2035961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhjhl94c/_remote_module_non_scriptable.py
2022-09-27T16:06:01.5076759Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:01.5288588Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42468
2022-09-27T16:06:01.5293640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42469
2022-09-27T16:06:02.9675121Z ok (7.161s)
2022-09-27T16:06:02.9675380Z 
2022-09-27T16:06:02.9675758Z ----------------------------------------------------------------------
2022-09-27T16:06:02.9676110Z Ran 1 test in 7.162s
2022-09-27T16:06:02.9676279Z 
2022-09-27T16:06:02.9676390Z OK
2022-09-27T16:06:02.9676535Z 
2022-09-27T16:06:02.9676675Z Generating XML reports...
2022-09-27T16:06:02.9712869Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160555.xml
2022-09-27T16:06:03.1963393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:03.1964152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:03.1965409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:03.1966182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:03.2137000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:03.2137816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:03.2139349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:03.2140164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:03.2205915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:03.2206667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:03.2209470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:03.2210305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:03.7254771Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:03.7276276Z 
2022-09-27T16:06:03.7276872Z Running tests...
2022-09-27T16:06:03.7277768Z ----------------------------------------------------------------------
2022-09-27T16:06:03.7300936Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:06:03.7301615Z 
2022-09-27T16:06:03.7302199Z ----------------------------------------------------------------------
2022-09-27T16:06:03.7302634Z Ran 1 test in 0.002s
2022-09-27T16:06:03.7302803Z 
2022-09-27T16:06:03.7302911Z OK (skipped=1)
2022-09-27T16:06:03.7303069Z 
2022-09-27T16:06:03.7303194Z Generating XML reports...
2022-09-27T16:06:03.7337782Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160603.xml
2022-09-27T16:06:03.7645074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:03.7645917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:03.7646954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:03.7647694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:03.7654053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:03.7654778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:03.7864262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:06:03.7864811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:06:03.7865594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:03.7866299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:03.8074162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:06:03.8074734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:06:03.8075488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:06:03.8076200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:06:05.0802762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:05.0803761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:05.0805347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:05.0806298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:05.2296191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ty0t1ov
2022-09-27T16:06:05.2297370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ty0t1ov/_remote_module_non_scriptable.py
2022-09-27T16:06:05.2309302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx23dm11w
2022-09-27T16:06:05.2311601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx23dm11w/_remote_module_non_scriptable.py
2022-09-27T16:06:05.5970295Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:06:05.5994267Z 
2022-09-27T16:06:05.5994856Z Running tests...
2022-09-27T16:06:05.5995743Z ----------------------------------------------------------------------
2022-09-27T16:06:05.6952802Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:05.6954268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:05.6955179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:05.6955832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:05.8394871Z ok (5.990s)
2022-09-27T16:06:05.8395296Z 
2022-09-27T16:06:05.8396016Z ----------------------------------------------------------------------
2022-09-27T16:06:05.8396656Z Ran 1 test in 5.990s
2022-09-27T16:06:05.8396992Z 
2022-09-27T16:06:05.8397180Z OK
2022-09-27T16:06:05.8397438Z 
2022-09-27T16:06:05.8397719Z Generating XML reports...
2022-09-27T16:06:05.8432275Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160559.xml
2022-09-27T16:06:06.2293244Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:06.2326721Z 
2022-09-27T16:06:06.2327077Z Running tests...
2022-09-27T16:06:06.2327528Z ----------------------------------------------------------------------
2022-09-27T16:06:06.2367210Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.004s)
2022-09-27T16:06:06.2367666Z 
2022-09-27T16:06:06.2368537Z ----------------------------------------------------------------------
2022-09-27T16:06:06.2369296Z Ran 1 test in 0.004s
2022-09-27T16:06:06.2369644Z 
2022-09-27T16:06:06.2369889Z OK (skipped=1)
2022-09-27T16:06:06.2370207Z 
2022-09-27T16:06:06.2370454Z Generating XML reports...
2022-09-27T16:06:06.2418674Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160606.xml
2022-09-27T16:06:07.4271195Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:07.4469348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42725
2022-09-27T16:06:07.4475493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42726
2022-09-27T16:06:08.0338419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:08.0339404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:08.0340636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:08.0341346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:08.2828684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:08.2829170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:08.2830830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:08.2831345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:08.5728870Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:06:08.5751237Z 
2022-09-27T16:06:08.5751500Z Running tests...
2022-09-27T16:06:08.5751944Z ----------------------------------------------------------------------
2022-09-27T16:06:08.7746024Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:08.7768554Z 
2022-09-27T16:06:08.7769015Z Running tests...
2022-09-27T16:06:08.7769704Z ----------------------------------------------------------------------
2022-09-27T16:06:08.7792034Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:06:08.7792650Z 
2022-09-27T16:06:08.7792939Z ----------------------------------------------------------------------
2022-09-27T16:06:08.7793280Z Ran 1 test in 0.002s
2022-09-27T16:06:08.7793844Z 
2022-09-27T16:06:08.7794001Z OK (skipped=1)
2022-09-27T16:06:08.7794168Z 
2022-09-27T16:06:08.7794296Z Generating XML reports...
2022-09-27T16:06:08.7827497Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160608.xml
2022-09-27T16:06:09.0863947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:09.0864583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:09.0867227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:09.0867731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:09.1013170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:09.1013695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:09.1016194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:09.1016690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:09.6180593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:09.6181447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:09.6182646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:09.6183753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:09.6188989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:09.6191272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:10.2663570Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:10.2869724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42835
2022-09-27T16:06:10.2874602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42836
2022-09-27T16:06:10.7735190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:10.7735725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:10.7737107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:10.7737850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:11.1219230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkogfjazd
2022-09-27T16:06:11.1219893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkogfjazd/_remote_module_non_scriptable.py
2022-09-27T16:06:11.1318252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnk48g49
2022-09-27T16:06:11.1322342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnk48g49/_remote_module_non_scriptable.py
2022-09-27T16:06:11.3530705Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:11.3553927Z 
2022-09-27T16:06:11.3554210Z Running tests...
2022-09-27T16:06:11.3554669Z ----------------------------------------------------------------------
2022-09-27T16:06:11.3582550Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-09-27T16:06:11.3583020Z 
2022-09-27T16:06:11.3583478Z ----------------------------------------------------------------------
2022-09-27T16:06:11.3583823Z Ran 1 test in 0.003s
2022-09-27T16:06:11.3583993Z 
2022-09-27T16:06:11.3584116Z OK (skipped=1)
2022-09-27T16:06:11.3584275Z 
2022-09-27T16:06:11.3584689Z Generating XML reports...
2022-09-27T16:06:11.3619923Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160611.xml
2022-09-27T16:06:11.9472356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:11.9473353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:11.9474546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:11.9475474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:12.0256643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:12.0257632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:12.0258817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:12.0259784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:12.6363202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:12.6363774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:12.6364573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:12.6365264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:12.6371565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:12.6372065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:12.6581701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:06:12.6582254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:06:12.6583026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:12.6583726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:12.6690133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:06:12.6690678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:06:12.6691709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:06:12.6692448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:06:12.7741250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:06:12.7741797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:06:13.3657014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:13.3657540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:13.3659549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:13.3660062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:13.4617474Z ok (7.862s)
2022-09-27T16:06:13.4617707Z 
2022-09-27T16:06:13.4618061Z ----------------------------------------------------------------------
2022-09-27T16:06:13.4618387Z Ran 1 test in 7.862s
2022-09-27T16:06:13.4618555Z 
2022-09-27T16:06:13.4618963Z OK
2022-09-27T16:06:13.4619105Z 
2022-09-27T16:06:13.4619239Z Generating XML reports...
2022-09-27T16:06:13.4654342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160605.xml
2022-09-27T16:06:13.8956498Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:13.8979873Z 
2022-09-27T16:06:13.8980098Z Running tests...
2022-09-27T16:06:13.8980849Z ----------------------------------------------------------------------
2022-09-27T16:06:13.9006334Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-09-27T16:06:13.9006790Z 
2022-09-27T16:06:13.9007325Z ----------------------------------------------------------------------
2022-09-27T16:06:13.9008085Z Ran 1 test in 0.003s
2022-09-27T16:06:13.9008409Z 
2022-09-27T16:06:13.9008566Z OK (skipped=1)
2022-09-27T16:06:13.9008725Z 
2022-09-27T16:06:13.9008863Z Generating XML reports...
2022-09-27T16:06:13.9042582Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160613.xml
2022-09-27T16:06:14.1301416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzza0phkl
2022-09-27T16:06:14.1302040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzza0phkl/_remote_module_non_scriptable.py
2022-09-27T16:06:14.1675535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfuqb8p00
2022-09-27T16:06:14.1677620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfuqb8p00/_remote_module_non_scriptable.py
2022-09-27T16:06:15.6304356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:15.6305222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:15.6306209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:15.6306990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:15.8296972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:15.8297480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:15.8300420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:15.8300917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:16.1262661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:06:16.1286235Z 
2022-09-27T16:06:16.1286766Z Running tests...
2022-09-27T16:06:16.1287231Z ----------------------------------------------------------------------
2022-09-27T16:06:16.3076588Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:16.3099156Z 
2022-09-27T16:06:16.3099581Z Running tests...
2022-09-27T16:06:16.3100110Z ----------------------------------------------------------------------
2022-09-27T16:06:16.3120690Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:06:16.3121287Z 
2022-09-27T16:06:16.3121620Z ----------------------------------------------------------------------
2022-09-27T16:06:16.3121942Z Ran 1 test in 0.002s
2022-09-27T16:06:16.3122109Z 
2022-09-27T16:06:16.3122221Z OK (skipped=1)
2022-09-27T16:06:16.3122378Z 
2022-09-27T16:06:16.3122504Z Generating XML reports...
2022-09-27T16:06:16.3156421Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160616.xml
2022-09-27T16:06:17.7441285Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:17.7606945Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.632s)
2022-09-27T16:06:17.7608605Z 
2022-09-27T16:06:17.7609181Z ----------------------------------------------------------------------
2022-09-27T16:06:17.7609849Z Ran 1 test in 1.632s
2022-09-27T16:06:17.7610168Z 
2022-09-27T16:06:17.7610377Z OK (skipped=1)
2022-09-27T16:06:17.7610679Z 
2022-09-27T16:06:17.7610903Z Generating XML reports...
2022-09-27T16:06:17.7641935Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160616.xml
2022-09-27T16:06:18.1691003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:18.1691981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:18.1693200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:18.1694144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:18.6617719Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:18.6641467Z 
2022-09-27T16:06:18.6641892Z Running tests...
2022-09-27T16:06:18.6642390Z ----------------------------------------------------------------------
2022-09-27T16:06:18.6666028Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:06:18.6666359Z 
2022-09-27T16:06:18.6667095Z ----------------------------------------------------------------------
2022-09-27T16:06:18.6667786Z Ran 1 test in 0.003s
2022-09-27T16:06:18.6668100Z 
2022-09-27T16:06:18.6668300Z OK (skipped=1)
2022-09-27T16:06:18.6668615Z 
2022-09-27T16:06:18.6668836Z Generating XML reports...
2022-09-27T16:06:18.6704204Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160618.xml
2022-09-27T16:06:19.8728433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:19.8728991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:19.8730761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:19.8731260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:20.3683627Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:06:20.3707218Z 
2022-09-27T16:06:20.3707778Z Running tests...
2022-09-27T16:06:20.3708240Z ----------------------------------------------------------------------
2022-09-27T16:06:20.5964888Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:20.5965521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:20.5968067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:20.5968560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:21.0875808Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:21.0898380Z 
2022-09-27T16:06:21.0898794Z Running tests...
2022-09-27T16:06:21.0899299Z ----------------------------------------------------------------------
2022-09-27T16:06:21.0920505Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:06:21.0921439Z 
2022-09-27T16:06:21.0922119Z ----------------------------------------------------------------------
2022-09-27T16:06:21.0922459Z Ran 1 test in 0.002s
2022-09-27T16:06:21.0922626Z 
2022-09-27T16:06:21.0922739Z OK (skipped=1)
2022-09-27T16:06:21.0922899Z 
2022-09-27T16:06:21.0923008Z Generating XML reports...
2022-09-27T16:06:21.0956804Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160621.xml
2022-09-27T16:06:21.9470061Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:21.9639786Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.593s)
2022-09-27T16:06:21.9641226Z 
2022-09-27T16:06:21.9641795Z ----------------------------------------------------------------------
2022-09-27T16:06:21.9642417Z Ran 1 test in 1.593s
2022-09-27T16:06:21.9642743Z 
2022-09-27T16:06:21.9642950Z OK (skipped=1)
2022-09-27T16:06:21.9643237Z 
2022-09-27T16:06:21.9643465Z Generating XML reports...
2022-09-27T16:06:21.9674911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160620.xml
2022-09-27T16:06:23.0061729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:23.0062251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:23.0064785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:23.0065267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:23.5457430Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:23.5480629Z 
2022-09-27T16:06:23.5480889Z Running tests...
2022-09-27T16:06:23.5481342Z ----------------------------------------------------------------------
2022-09-27T16:06:24.0758978Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:24.0759606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:24.0760997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:24.0761476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:24.5920481Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:06:24.5944850Z 
2022-09-27T16:06:24.5945527Z Running tests...
2022-09-27T16:06:24.5946044Z ----------------------------------------------------------------------
2022-09-27T16:06:24.7156442Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... ok (16.140s)
2022-09-27T16:06:24.7156825Z 
2022-09-27T16:06:24.7157437Z ----------------------------------------------------------------------
2022-09-27T16:06:24.7157782Z Ran 1 test in 16.140s
2022-09-27T16:06:24.7157954Z 
2022-09-27T16:06:24.7158049Z OK
2022-09-27T16:06:24.7158187Z 
2022-09-27T16:06:24.7158316Z Generating XML reports...
2022-09-27T16:06:24.7193456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160608.xml
2022-09-27T16:06:25.1830069Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:25.2029809Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43206
2022-09-27T16:06:25.2036568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43207
2022-09-27T16:06:26.3238871Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:26.3436712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43302
2022-09-27T16:06:26.3443189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43303
2022-09-27T16:06:26.8217626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:26.8218134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:26.8220756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:26.8221241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:26.8238337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:26.8238829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:26.8243101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:26.8243580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:26.8279480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:26.8279952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:26.8284253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:26.8284726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:27.3845585Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:06:27.3867742Z 
2022-09-27T16:06:27.3868067Z Running tests...
2022-09-27T16:06:27.3868519Z ----------------------------------------------------------------------
2022-09-27T16:06:27.4049271Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:27.4103420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:27.4104246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:27.4150729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:27.4157497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:27.5118444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:27.9102911Z ok (4.362s)
2022-09-27T16:06:27.9103134Z 
2022-09-27T16:06:27.9103786Z ----------------------------------------------------------------------
2022-09-27T16:06:27.9104161Z Ran 1 test in 4.362s
2022-09-27T16:06:27.9104312Z 
2022-09-27T16:06:27.9104418Z OK
2022-09-27T16:06:27.9104566Z 
2022-09-27T16:06:27.9104703Z Generating XML reports...
2022-09-27T16:06:27.9139760Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160623.xml
2022-09-27T16:06:28.0174888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:28.0175601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:28.0176440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:28.0176915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:28.0209546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:28.0210032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:28.0213673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:28.0214520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:28.5423550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:28.5568593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:28.5569432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:28.5625956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:28.5632990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:28.6583411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:29.0453733Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:29.0902779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43415
2022-09-27T16:06:29.0994247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43416
2022-09-27T16:06:29.9876551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlrg4v2a
2022-09-27T16:06:29.9877465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlrg4v2a/_remote_module_non_scriptable.py
2022-09-27T16:06:30.0841317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:30.0841846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:30.0843001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:30.0843477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:30.1125203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfj83bud
2022-09-27T16:06:30.1126554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfj83bud/_remote_module_non_scriptable.py
2022-09-27T16:06:30.5789772Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:30.5812369Z 
2022-09-27T16:06:30.5812610Z Running tests...
2022-09-27T16:06:30.5813055Z ----------------------------------------------------------------------
2022-09-27T16:06:30.7825813Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:30.7826444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:30.7827239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:30.7827725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:30.7828321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:30.7828816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:30.7829879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:30.7830517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:31.3411828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:31.3412384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:31.3413225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:31.3413941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:31.3521329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:31.4427611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:32.2177603Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:32.2377175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43506
2022-09-27T16:06:32.2382494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43507
2022-09-27T16:06:32.2570515Z ok (7.662s)
2022-09-27T16:06:32.2570731Z 
2022-09-27T16:06:32.2571117Z ----------------------------------------------------------------------
2022-09-27T16:06:32.2571440Z Ran 1 test in 7.662s
2022-09-27T16:06:32.2571637Z 
2022-09-27T16:06:32.2571735Z OK
2022-09-27T16:06:32.2571874Z 
2022-09-27T16:06:32.2572009Z Generating XML reports...
2022-09-27T16:06:32.2608681Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160624.xml
2022-09-27T16:06:32.8117400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqri0kpm
2022-09-27T16:06:32.8118612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqri0kpm/_remote_module_non_scriptable.py
2022-09-27T16:06:32.9122302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31bp2t9g
2022-09-27T16:06:32.9124073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31bp2t9g/_remote_module_non_scriptable.py
2022-09-27T16:06:33.9687409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:33.9687952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:33.9688768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:33.9689251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:33.9931910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:33.9932412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:33.9935089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:33.9935563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:34.3109454Z ok (6.924s)
2022-09-27T16:06:34.3109660Z 
2022-09-27T16:06:34.3110060Z ----------------------------------------------------------------------
2022-09-27T16:06:34.3110387Z Ran 1 test in 6.924s
2022-09-27T16:06:34.3110556Z 
2022-09-27T16:06:34.3110863Z OK
2022-09-27T16:06:34.3111018Z 
2022-09-27T16:06:34.3111156Z Generating XML reports...
2022-09-27T16:06:34.3146183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160627.xml
2022-09-27T16:06:34.5013163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:34.5013667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:34.5015597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:34.5016095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:34.5068019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:34.5068939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:34.5070204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:34.5070950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:34.5078696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:34.5079328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:34.9449514Z ok (4.363s)
2022-09-27T16:06:34.9449778Z 
2022-09-27T16:06:34.9450504Z ----------------------------------------------------------------------
2022-09-27T16:06:34.9451208Z Ran 1 test in 4.364s
2022-09-27T16:06:34.9451428Z 
2022-09-27T16:06:34.9451532Z OK
2022-09-27T16:06:34.9451671Z 
2022-09-27T16:06:34.9451789Z Generating XML reports...
2022-09-27T16:06:34.9487494Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160630.xml
2022-09-27T16:06:35.0039272Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:06:35.0061105Z 
2022-09-27T16:06:35.0061621Z Running tests...
2022-09-27T16:06:35.0062557Z ----------------------------------------------------------------------
2022-09-27T16:06:36.4010830Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:36.4011563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:36.4012859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:36.4013403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:36.7321492Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:36.7517243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43678
2022-09-27T16:06:36.7523942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43679
2022-09-27T16:06:36.9061213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:06:36.9083534Z 
2022-09-27T16:06:36.9083949Z Running tests...
2022-09-27T16:06:36.9084888Z ----------------------------------------------------------------------
2022-09-27T16:06:37.0004942Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:37.0005567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:37.0008307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:37.0009060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:37.5955918Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:37.5977972Z 
2022-09-27T16:06:37.5978408Z Running tests...
2022-09-27T16:06:37.5979379Z ----------------------------------------------------------------------
2022-09-27T16:06:37.5998323Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:06:37.5998898Z 
2022-09-27T16:06:37.5999493Z ----------------------------------------------------------------------
2022-09-27T16:06:37.6000494Z Ran 1 test in 0.002s
2022-09-27T16:06:37.6000757Z 
2022-09-27T16:06:37.6000874Z OK (skipped=1)
2022-09-27T16:06:37.6001043Z 
2022-09-27T16:06:37.6001175Z Generating XML reports...
2022-09-27T16:06:37.6035357Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160637.xml
2022-09-27T16:06:38.3778259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:38.3778829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:38.3780131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:38.3781140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:38.4690295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:38.4691162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:38.4692418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:38.4693167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:38.5777409Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:38.6449717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43779
2022-09-27T16:06:38.6567398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43780
2022-09-27T16:06:38.9197581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:39.0237647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:39.0238500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:39.0311024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:39.0318572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:39.0322326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:06:39.1250427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:39.1253149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:06:39.1253924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:39.1336405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:39.6414259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:39.6414775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:39.6417567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:39.6418076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:39.6593582Z ok (4.653s)
2022-09-27T16:06:39.6595771Z 
2022-09-27T16:06:39.6596127Z ----------------------------------------------------------------------
2022-09-27T16:06:39.6596457Z Ran 1 test in 4.653s
2022-09-27T16:06:39.6596638Z 
2022-09-27T16:06:39.6596736Z OK
2022-09-27T16:06:39.6596875Z 
2022-09-27T16:06:39.6597006Z Generating XML reports...
2022-09-27T16:06:39.6632783Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160634.xml
2022-09-27T16:06:40.1942297Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:40.1965183Z 
2022-09-27T16:06:40.1965492Z Running tests...
2022-09-27T16:06:40.1965943Z ----------------------------------------------------------------------
2022-09-27T16:06:40.1984586Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:06:40.1985068Z 
2022-09-27T16:06:40.1985504Z ----------------------------------------------------------------------
2022-09-27T16:06:40.1985849Z Ran 1 test in 0.002s
2022-09-27T16:06:40.1986020Z 
2022-09-27T16:06:40.1986133Z OK (skipped=1)
2022-09-27T16:06:40.1986592Z 
2022-09-27T16:06:40.1986723Z Generating XML reports...
2022-09-27T16:06:40.2020766Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160640.xml
2022-09-27T16:06:40.3109976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:40.3110482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:40.3112967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:40.3113450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:40.3541519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:40.3542016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:40.3544285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:40.3544776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:40.9410982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:40.9412072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:40.9413571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:40.9415015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:40.9418420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:40.9419871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:41.8312404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:41.8312953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:41.8314483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:41.8314957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:42.1958124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:42.1958631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:42.1961147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:42.1961690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:42.3020471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfpourjk
2022-09-27T16:06:42.3021332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfpourjk/_remote_module_non_scriptable.py
2022-09-27T16:06:42.3423426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01loakjj
2022-09-27T16:06:42.3425099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01loakjj/_remote_module_non_scriptable.py
2022-09-27T16:06:42.3468114Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:06:42.3492980Z 
2022-09-27T16:06:42.3493386Z Running tests...
2022-09-27T16:06:42.3493877Z ----------------------------------------------------------------------
2022-09-27T16:06:42.7030459Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:42.7053101Z 
2022-09-27T16:06:42.7053521Z Running tests...
2022-09-27T16:06:42.7054033Z ----------------------------------------------------------------------
2022-09-27T16:06:43.7682575Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... ok (6.859s)
2022-09-27T16:06:43.7683156Z 
2022-09-27T16:06:43.7683688Z ----------------------------------------------------------------------
2022-09-27T16:06:43.7684025Z Ran 1 test in 6.860s
2022-09-27T16:06:43.7684196Z 
2022-09-27T16:06:43.7684292Z OK
2022-09-27T16:06:43.7684428Z 
2022-09-27T16:06:43.7684559Z Generating XML reports...
2022-09-27T16:06:43.7719511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160636.xml
2022-09-27T16:06:44.0672722Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:44.0872644Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43939
2022-09-27T16:06:44.0877345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43940
2022-09-27T16:06:44.3983048Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:44.4148219Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.709s)
2022-09-27T16:06:44.4148816Z 
2022-09-27T16:06:44.4149097Z ----------------------------------------------------------------------
2022-09-27T16:06:44.4149442Z Ran 1 test in 1.709s
2022-09-27T16:06:44.4149612Z 
2022-09-27T16:06:44.4149708Z OK (skipped=1)
2022-09-27T16:06:44.4149869Z 
2022-09-27T16:06:44.4149999Z Generating XML reports...
2022-09-27T16:06:44.4183487Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160642.xml
2022-09-27T16:06:45.7460238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:45.7460812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:45.7461776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:45.7462272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:45.8410911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:45.8411477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:45.8412285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:45.8412780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:46.0098060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:46.0099015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:46.0100477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:46.0100978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:46.2449092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:46.3566253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:46.3567068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:46.3663951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:46.3670912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:46.3674703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:06:46.4576400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:46.4580154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:06:46.4580877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:46.4587081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:06:46.4938350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:06:46.4961184Z 
2022-09-27T16:06:46.4961758Z Running tests...
2022-09-27T16:06:46.4962336Z ----------------------------------------------------------------------
2022-09-27T16:06:46.5274735Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:46.5275759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:46.5276675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:46.5277170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:46.8946123Z ok (4.545s)
2022-09-27T16:06:46.8946350Z 
2022-09-27T16:06:46.8946876Z ----------------------------------------------------------------------
2022-09-27T16:06:46.8947331Z Ran 1 test in 4.545s
2022-09-27T16:06:46.8947505Z 
2022-09-27T16:06:46.8947601Z OK
2022-09-27T16:06:46.8947746Z 
2022-09-27T16:06:46.8947883Z Generating XML reports...
2022-09-27T16:06:46.8981900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160642.xml
2022-09-27T16:06:47.0391376Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:47.0415162Z 
2022-09-27T16:06:47.0415541Z Running tests...
2022-09-27T16:06:47.0416039Z ----------------------------------------------------------------------
2022-09-27T16:06:48.1499467Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:48.1693663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44116
2022-09-27T16:06:48.1699096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44117
2022-09-27T16:06:48.6856968Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:48.7056193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44119
2022-09-27T16:06:48.7062407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44120
2022-09-27T16:06:49.1104284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:49.1104846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:49.1106269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:49.1106768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:49.6839636Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:06:49.6863956Z 
2022-09-27T16:06:49.6864282Z Running tests...
2022-09-27T16:06:49.6864744Z ----------------------------------------------------------------------
2022-09-27T16:06:49.8955161Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:49.8955791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:49.8956970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:49.8957782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:49.9106345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:49.9106837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:49.9109991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:49.9110480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:50.4216996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:50.4217525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:50.4218557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:50.4219060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:50.4382010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:50.4382473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:50.4384946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:50.4385434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:50.4531585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:50.4532111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:50.4532876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:50.4533587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:50.4541515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:50.4541994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:50.9512789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:50.9624292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:50.9625146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:50.9717692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:50.9723728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:51.0638594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:51.3954843Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:51.4170698Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44266
2022-09-27T16:06:51.4175945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44267
2022-09-27T16:06:51.8528135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi1behjh
2022-09-27T16:06:51.8528764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi1behjh/_remote_module_non_scriptable.py
2022-09-27T16:06:51.8678840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2mtmx5q
2022-09-27T16:06:51.8680456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2mtmx5q/_remote_module_non_scriptable.py
2022-09-27T16:06:52.3913217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lx5q4rx
2022-09-27T16:06:52.3914803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lx5q4rx/_remote_module_non_scriptable.py
2022-09-27T16:06:52.4924851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdimvf85g
2022-09-27T16:06:52.4925960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdimvf85g/_remote_module_non_scriptable.py
2022-09-27T16:06:52.5194668Z [1664294812.518903] [28fd6574525b:44119:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:06:52.5195859Z [1664294812.518903] [28fd6574525b:44119:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:06:52.5282037Z [1664294812.527766] [28fd6574525b:44120:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:06:52.5282608Z [1664294812.527766] [28fd6574525b:44120:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:06:52.9808067Z ok (6.484s)
2022-09-27T16:06:52.9808511Z 
2022-09-27T16:06:52.9809225Z ----------------------------------------------------------------------
2022-09-27T16:06:52.9809892Z Ran 1 test in 6.484s
2022-09-27T16:06:52.9810230Z 
2022-09-27T16:06:52.9810399Z OK
2022-09-27T16:06:52.9810677Z 
2022-09-27T16:06:52.9810948Z Generating XML reports...
2022-09-27T16:06:52.9844785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160646.xml
2022-09-27T16:06:53.1042464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:53.1043363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:53.1044564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:53.1045505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:53.1058240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:53.1059115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:53.1062632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:53.1063749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:53.6399716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:53.6537091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:53.6538819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:53.6602716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:53.6609703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:06:53.7549089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:54.0179734Z ok (6.976s)
2022-09-27T16:06:54.0179934Z 
2022-09-27T16:06:54.0180334Z ----------------------------------------------------------------------
2022-09-27T16:06:54.0180680Z Ran 1 test in 6.976s
2022-09-27T16:06:54.0180848Z 
2022-09-27T16:06:54.0180943Z OK
2022-09-27T16:06:54.0181079Z 
2022-09-27T16:06:54.0181207Z Generating XML reports...
2022-09-27T16:06:54.0217535Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160647.xml
2022-09-27T16:06:55.1248534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1tmkd3q
2022-09-27T16:06:55.1249816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1tmkd3q/_remote_module_non_scriptable.py
2022-09-27T16:06:55.1531432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps66964g1
2022-09-27T16:06:55.1533371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps66964g1/_remote_module_non_scriptable.py
2022-09-27T16:06:55.1952683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:55.1953716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:55.1958545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:55.1959336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:55.6823731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:06:55.6824294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:06:55.6886431Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:06:55.6888038Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:06:55.7124474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:06:55.7150670Z 
2022-09-27T16:06:55.7150826Z Running tests...
2022-09-27T16:06:55.7151610Z ----------------------------------------------------------------------
2022-09-27T16:06:56.1794446Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:56.1795039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:56.1796980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:56.1797744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:56.3296985Z ok (6.643s)
2022-09-27T16:06:56.3297186Z 
2022-09-27T16:06:56.3297606Z ----------------------------------------------------------------------
2022-09-27T16:06:56.3297964Z Ran 1 test in 6.643s
2022-09-27T16:06:56.3298130Z 
2022-09-27T16:06:56.3298226Z OK
2022-09-27T16:06:56.3298362Z 
2022-09-27T16:06:56.3298486Z Generating XML reports...
2022-09-27T16:06:56.3334217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160649.xml
2022-09-27T16:06:56.6825410Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:06:56.6847518Z 
2022-09-27T16:06:56.6847665Z Running tests...
2022-09-27T16:06:56.6848097Z ----------------------------------------------------------------------
2022-09-27T16:06:57.5393368Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:57.5624088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44465
2022-09-27T16:06:57.5630140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44466
2022-09-27T16:06:58.4067370Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:06:58.4262795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44530
2022-09-27T16:06:58.4269094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44531
2022-09-27T16:06:58.4991578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:58.4992083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:58.4994152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:58.4994654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:58.9905693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:06:58.9928229Z 
2022-09-27T16:06:58.9928462Z Running tests...
2022-09-27T16:06:58.9928911Z ----------------------------------------------------------------------
2022-09-27T16:06:59.2649900Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:59.2651050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:59.2652584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:59.2653545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:59.3470040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:06:59.3470937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:06:59.3472518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:06:59.3473501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:06:59.8899289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:06:59.8900428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:06:59.8901880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:59.8903293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:06:59.9007758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:06:59.9915401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:00.0590368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:00.0591346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:00.0592424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:00.0593340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:00.0756166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:00.0757142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:00.0758827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:00.0759818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:00.5993076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:00.6209977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:00.6211614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:00.6298752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:00.6305285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:00.6553913Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:00.6758526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44613
2022-09-27T16:07:00.6765279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44614
2022-09-27T16:07:00.7225365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:01.2922567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphx5w029z
2022-09-27T16:07:01.2923865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphx5w029z/_remote_module_non_scriptable.py
2022-09-27T16:07:01.3955604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyfvaqu6
2022-09-27T16:07:01.3957546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyfvaqu6/_remote_module_non_scriptable.py
2022-09-27T16:07:01.8556241Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:01.8609662Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:01.9396616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:01.9397185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:02.3551304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:02.3552269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:02.3553469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:02.3554384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:02.3647956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:02.3648869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:02.3650972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:02.3651887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:02.5749228Z ok (6.860s)
2022-09-27T16:07:02.5749442Z 
2022-09-27T16:07:02.5749882Z ----------------------------------------------------------------------
2022-09-27T16:07:02.5750206Z Ran 1 test in 6.860s
2022-09-27T16:07:02.5750385Z 
2022-09-27T16:07:02.5750479Z OK
2022-09-27T16:07:02.5750891Z 
2022-09-27T16:07:02.5751026Z Generating XML reports...
2022-09-27T16:07:02.5787038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160655.xml
2022-09-27T16:07:02.8994511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:02.9119346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:02.9120633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:02.9198335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:02.9205949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:03.0135032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:03.2535283Z [1664294823.253007] [28fd6574525b:44530:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:03.2535834Z [1664294823.253007] [28fd6574525b:44530:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:03.2553243Z [1664294823.255015] [28fd6574525b:44531:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:03.2553737Z [1664294823.255015] [28fd6574525b:44531:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:04.2396277Z ok (7.554s)
2022-09-27T16:07:04.2396502Z 
2022-09-27T16:07:04.2396898Z ----------------------------------------------------------------------
2022-09-27T16:07:04.2397256Z Ran 1 test in 7.555s
2022-09-27T16:07:04.2397427Z 
2022-09-27T16:07:04.2397511Z OK
2022-09-27T16:07:04.2397647Z 
2022-09-27T16:07:04.2397800Z Generating XML reports...
2022-09-27T16:07:04.2562122Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160656.xml
2022-09-27T16:07:04.4220390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxo_nw0d
2022-09-27T16:07:04.4221402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxo_nw0d/_remote_module_non_scriptable.py
2022-09-27T16:07:04.5024059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vs8nx0i
2022-09-27T16:07:04.5024718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vs8nx0i/_remote_module_non_scriptable.py
2022-09-27T16:07:04.8427000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:04.8428266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:04.8429897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:04.8430784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:05.0426924Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:05.0582419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:05.0582965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:05.3849122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:07:05.3872196Z 
2022-09-27T16:07:05.3872551Z Running tests...
2022-09-27T16:07:05.3873003Z ----------------------------------------------------------------------
2022-09-27T16:07:05.6884778Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... ok (6.695s)
2022-09-27T16:07:05.6885092Z 
2022-09-27T16:07:05.6885494Z ----------------------------------------------------------------------
2022-09-27T16:07:05.6885831Z Ran 1 test in 6.695s
2022-09-27T16:07:05.6885980Z 
2022-09-27T16:07:05.6886074Z OK
2022-09-27T16:07:05.6886209Z 
2022-09-27T16:07:05.6886338Z Generating XML reports...
2022-09-27T16:07:05.6921668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160658.xml
2022-09-27T16:07:06.5576096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:06.5576667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:06.5577850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:06.5578338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:07.1198105Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:07:07.1220855Z 
2022-09-27T16:07:07.1221112Z Running tests...
2022-09-27T16:07:07.1221581Z ----------------------------------------------------------------------
2022-09-27T16:07:07.1865438Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:07.2074631Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44815
2022-09-27T16:07:07.2079475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44816
2022-09-27T16:07:07.8099618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:07.8100628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:07.8101934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:07.8102465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:08.3657130Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:08.3679628Z 
2022-09-27T16:07:08.3679865Z Running tests...
2022-09-27T16:07:08.3680829Z ----------------------------------------------------------------------
2022-09-27T16:07:08.7253507Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:08.7451977Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44882
2022-09-27T16:07:08.7456392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44883
2022-09-27T16:07:08.8790061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:08.8790701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:08.8791977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:08.8792465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:08.9159445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:08.9160676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:08.9161953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:08.9162443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:09.5392771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:09.5394213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:09.5395666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:09.5397018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:09.5400780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:09.5401746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:10.0659412Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:10.0869922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44959
2022-09-27T16:07:10.0875017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44960
2022-09-27T16:07:10.3900948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:10.3901952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:10.3903163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:10.3904094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:10.4701281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:10.4702264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:10.4703449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:10.4704382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:10.9803933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpainti789
2022-09-27T16:07:10.9804571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpainti789/_remote_module_non_scriptable.py
2022-09-27T16:07:10.9836481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp374sy6ok
2022-09-27T16:07:10.9839398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp374sy6ok/_remote_module_non_scriptable.py
2022-09-27T16:07:11.0024500Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:07:11.0025375Z   warnings.warn(
2022-09-27T16:07:11.0026438Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:07:11.0027193Z   warnings.warn(
2022-09-27T16:07:11.0490826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:11.1066152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:11.1067354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:11.1099388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:11.1105766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:11.2079705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:11.8055087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:11.8056089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:11.8057562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:11.8058527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:11.8989139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:11.8990068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:11.8993421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:11.8994377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:12.0189056Z ok (6.631s)
2022-09-27T16:07:12.0189459Z 
2022-09-27T16:07:12.0190116Z ----------------------------------------------------------------------
2022-09-27T16:07:12.0190772Z Ran 1 test in 6.632s
2022-09-27T16:07:12.0191107Z 
2022-09-27T16:07:12.0191273Z OK
2022-09-27T16:07:12.0191520Z 
2022-09-27T16:07:12.0191785Z Generating XML reports...
2022-09-27T16:07:12.0228211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160705.xml
2022-09-27T16:07:12.3507126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:12.4617214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:12.4618736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:12.4624499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:12.4631626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:12.5629225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:12.9946515Z ok (4.626s)
2022-09-27T16:07:12.9946915Z 
2022-09-27T16:07:12.9947592Z ----------------------------------------------------------------------
2022-09-27T16:07:12.9948232Z Ran 1 test in 4.627s
2022-09-27T16:07:12.9948551Z 
2022-09-27T16:07:12.9948710Z OK
2022-09-27T16:07:12.9948983Z 
2022-09-27T16:07:12.9949232Z Generating XML reports...
2022-09-27T16:07:12.9984624Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160708.xml
2022-09-27T16:07:13.7398006Z [1664294833.739324] [28fd6574525b:44882:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:13.7398578Z [1664294833.739324] [28fd6574525b:44882:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:13.7426948Z [1664294833.742367] [28fd6574525b:44883:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:13.7427667Z [1664294833.742367] [28fd6574525b:44883:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:14.3471194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:14.3471771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:14.3473576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:14.3474511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:14.8616114Z ok (7.739s)
2022-09-27T16:07:14.8616339Z 
2022-09-27T16:07:14.8616738Z ----------------------------------------------------------------------
2022-09-27T16:07:14.8617090Z Ran 1 test in 7.739s
2022-09-27T16:07:14.8617241Z 
2022-09-27T16:07:14.8617339Z OK
2022-09-27T16:07:14.8617476Z 
2022-09-27T16:07:14.8617612Z Generating XML reports...
2022-09-27T16:07:14.8653239Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160707.xml
2022-09-27T16:07:14.8686596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:07:14.8711029Z 
2022-09-27T16:07:14.8711320Z Running tests...
2022-09-27T16:07:14.8711784Z ----------------------------------------------------------------------
2022-09-27T16:07:15.1284122Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:15.1284745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:15.1286842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:15.1287351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:15.6483613Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:15.6505349Z 
2022-09-27T16:07:15.6505573Z Running tests...
2022-09-27T16:07:15.6506044Z ----------------------------------------------------------------------
2022-09-27T16:07:15.6524705Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:07:15.6526353Z 
2022-09-27T16:07:15.6527064Z ----------------------------------------------------------------------
2022-09-27T16:07:15.6527456Z Ran 1 test in 0.002s
2022-09-27T16:07:15.6527627Z 
2022-09-27T16:07:15.6527762Z OK (skipped=1)
2022-09-27T16:07:15.6527933Z 
2022-09-27T16:07:15.6528065Z Generating XML reports...
2022-09-27T16:07:15.6562480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160715.xml
2022-09-27T16:07:16.5131036Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:16.5332298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45152
2022-09-27T16:07:16.5338098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45153
2022-09-27T16:07:16.9703742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:16.9704698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:16.9706711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:16.9707711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:17.5389055Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:07:17.5413977Z 
2022-09-27T16:07:17.5414375Z Running tests...
2022-09-27T16:07:17.5415232Z ----------------------------------------------------------------------
2022-09-27T16:07:17.7612810Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:17.7613992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:17.7615115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:17.7616124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:18.2257245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:18.2258684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:18.2259888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:18.2260814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:18.2558804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:18.2559793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:18.2561665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:18.2562597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:18.2826375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:18.2850659Z 
2022-09-27T16:07:18.2851393Z Running tests...
2022-09-27T16:07:18.2852266Z ----------------------------------------------------------------------
2022-09-27T16:07:18.2875739Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:07:18.2876360Z 
2022-09-27T16:07:18.2876898Z ----------------------------------------------------------------------
2022-09-27T16:07:18.2877526Z Ran 1 test in 0.003s
2022-09-27T16:07:18.2877841Z 
2022-09-27T16:07:18.2878048Z OK (skipped=1)
2022-09-27T16:07:18.2878333Z 
2022-09-27T16:07:18.2878580Z Generating XML reports...
2022-09-27T16:07:18.2913699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160718.xml
2022-09-27T16:07:18.7877302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:18.7878378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:18.7879869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:18.7881716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:18.7985962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:18.8892559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:19.1350309Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:19.1557588Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45264
2022-09-27T16:07:19.1563707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45265
2022-09-27T16:07:20.2625388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3e6fqp6n
2022-09-27T16:07:20.2626107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3e6fqp6n/_remote_module_non_scriptable.py
2022-09-27T16:07:20.3237314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprplcum8z
2022-09-27T16:07:20.3238479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprplcum8z/_remote_module_non_scriptable.py
2022-09-27T16:07:20.4811265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:20.4811794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:20.4813343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:20.4813967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:20.6248662Z STAGE:2022-09-27 16:07:20 45152:45152 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:07:20 45153:45153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:20.6249399Z 
2022-09-27T16:07:20.9100542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:20.9101083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:20.9101921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:20.9102397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:20.9792819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:20.9793335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:20.9794341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:20.9795024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:21.0165127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:21.0189448Z 
2022-09-27T16:07:21.0189981Z Running tests...
2022-09-27T16:07:21.0190531Z ----------------------------------------------------------------------
2022-09-27T16:07:21.1555409Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:21.1556047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:21.1646333Z STAGE:2022-09-27 16:07:21 45152:45152 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:21.1647201Z STAGE:2022-09-27 16:07:21 45153:45153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:21.1893031Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:21.1894943Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:21.1941787Z STAGE:2022-09-27 16:07:21 45152:45152 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:21.1942432Z STAGE:2022-09-27 16:07:21 45153:45153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:21.1986389Z STAGE:2022-09-27 16:07:21 45152:45152 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:21.1987022Z STAGE:2022-09-27 16:07:21 45153:45153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:21.4438080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:21.5225782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:21.5226684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:21.5251188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:21.5257848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:21.6238804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:21.8457808Z ok (6.974s)
2022-09-27T16:07:21.8458076Z 
2022-09-27T16:07:21.8458470Z ----------------------------------------------------------------------
2022-09-27T16:07:21.8458811Z Ran 1 test in 6.975s
2022-09-27T16:07:21.8458976Z 
2022-09-27T16:07:21.8459068Z OK
2022-09-27T16:07:21.8459205Z 
2022-09-27T16:07:21.8459322Z Generating XML reports...
2022-09-27T16:07:21.8495699Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160714.xml
2022-09-27T16:07:22.8311649Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:22.8520058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45377
2022-09-27T16:07:22.8525083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45378
2022-09-27T16:07:22.9892349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qwm0z0y
2022-09-27T16:07:22.9893322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qwm0z0y/_remote_module_non_scriptable.py
2022-09-27T16:07:23.0486585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoya6wnk3
2022-09-27T16:07:23.0487703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoya6wnk3/_remote_module_non_scriptable.py
2022-09-27T16:07:23.0752734Z [1664294843.074887] [28fd6574525b:45264:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:23.0753223Z [1664294843.074887] [28fd6574525b:45264:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:23.0834963Z [1664294843.083168] [28fd6574525b:45265:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:23.0835452Z [1664294843.083168] [28fd6574525b:45265:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:23.9750735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:23.9754521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:23.9768612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:23.9773808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:24.2094856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:24.2095511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:24.2096973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:24.2097488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:24.5103945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:24.5104857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:24.5105958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:24.5106819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:24.5659349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:24.5660234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:24.5662283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:24.5663124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:24.6686498Z ok (7.127s)
2022-09-27T16:07:24.6686731Z 
2022-09-27T16:07:24.6687098Z ----------------------------------------------------------------------
2022-09-27T16:07:24.6687444Z Ran 1 test in 7.127s
2022-09-27T16:07:24.6687607Z 
2022-09-27T16:07:24.6687705Z OK
2022-09-27T16:07:24.6687851Z 
2022-09-27T16:07:24.6687989Z Generating XML reports...
2022-09-27T16:07:24.6722776Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160717.xml
2022-09-27T16:07:24.7320847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:07:24.7345038Z 
2022-09-27T16:07:24.7345300Z Running tests...
2022-09-27T16:07:24.7345741Z ----------------------------------------------------------------------
2022-09-27T16:07:25.0374793Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:25.1218833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:25.1219654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:25.1287168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:25.1294095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:25.2232236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:26.4412469Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:26.4620832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45530
2022-09-27T16:07:26.4625864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45531
2022-09-27T16:07:26.7362540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:26.7363062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:26.7364537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:26.7365030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:26.7628615Z STAGE:2022-09-27 16:07:26 45378:45378 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:26.8559119Z STAGE:2022-09-27 16:07:26 45377:45377 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:27.3045934Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:07:27.3068353Z 
2022-09-27T16:07:27.3068936Z Running tests...
2022-09-27T16:07:27.3069394Z ----------------------------------------------------------------------
2022-09-27T16:07:28.1076435Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:28.1077093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:28.1077700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:28.1078186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:28.1367135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:28.1367622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:28.1369319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:28.1369806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:28.2271403Z STAGE:2022-09-27 16:07:28 45377:45377 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:28.2329882Z STAGE:2022-09-27 16:07:28 45378:45378 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:28.4043411Z STAGE:2022-09-27 16:07:28 45378:45378 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:28.4050205Z STAGE:2022-09-27 16:07:28 45377:45377 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:28.4144798Z STAGE:2022-09-27 16:07:28 45378:45378 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:28.4198442Z STAGE:2022-09-27 16:07:28 45377:45377 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:28.6898992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:28.6899575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:28.6900387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:28.6901080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:28.7009824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:28.7913501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:28.9084483Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:28.9291359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45626
2022-09-27T16:07:28.9296967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45627
2022-09-27T16:07:29.2679434Z ok (8.249s)
2022-09-27T16:07:29.2679800Z 
2022-09-27T16:07:29.2680605Z ----------------------------------------------------------------------
2022-09-27T16:07:29.2680965Z Ran 1 test in 8.249s
2022-09-27T16:07:29.2681127Z 
2022-09-27T16:07:29.2681223Z OK
2022-09-27T16:07:29.2681359Z 
2022-09-27T16:07:29.2681498Z Generating XML reports...
2022-09-27T16:07:29.2738439Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160721.xml
2022-09-27T16:07:30.1713075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8xc7dby
2022-09-27T16:07:30.1714059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8xc7dby/_remote_module_non_scriptable.py
2022-09-27T16:07:30.2820182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqjvgm2l
2022-09-27T16:07:30.2821629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqjvgm2l/_remote_module_non_scriptable.py
2022-09-27T16:07:30.5853451Z STAGE:2022-09-27 16:07:30 45531:45531 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:30.6321798Z STAGE:2022-09-27 16:07:30 45530:45530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:30.6866855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:30.6867375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:30.6867937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:30.6868396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:30.6869278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:30.6869766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:30.6870737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:30.6871219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:31.2039197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:31.2039738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:31.2135626Z STAGE:2022-09-27 16:07:31 45531:45531 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:31.2136255Z STAGE:2022-09-27 16:07:31 45530:45530 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:31.2255868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:31.2281695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:31.2282435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:31.2359325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:31.2365682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:31.2423327Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:31.2424928Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:31.2471046Z STAGE:2022-09-27 16:07:31 45530:45530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:31.2471626Z STAGE:2022-09-27 16:07:31 45531:45531 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:07:31.2517257Z STAGE:2022-09-27 16:07:31 45531:45531 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:07:31 45530:45530 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:07:31.2517671Z 
2022-09-27T16:07:31.3297866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:31.5227851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:31.5228403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:31.5230261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:31.5230735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:31.8752445Z ok (7.140s)
2022-09-27T16:07:31.8752667Z 
2022-09-27T16:07:31.8753058Z ----------------------------------------------------------------------
2022-09-27T16:07:31.8753376Z Ran 1 test in 7.141s
2022-09-27T16:07:31.8753545Z 
2022-09-27T16:07:31.8753644Z OK
2022-09-27T16:07:31.8753777Z 
2022-09-27T16:07:31.8753921Z Generating XML reports...
2022-09-27T16:07:31.8789485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160724.xml
2022-09-27T16:07:32.0644273Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:32.0668017Z 
2022-09-27T16:07:32.0668464Z Running tests...
2022-09-27T16:07:32.0669003Z ----------------------------------------------------------------------
2022-09-27T16:07:32.0687944Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:07:32.0688388Z 
2022-09-27T16:07:32.0688925Z ----------------------------------------------------------------------
2022-09-27T16:07:32.0689348Z Ran 1 test in 0.002s
2022-09-27T16:07:32.0689518Z 
2022-09-27T16:07:32.0689629Z OK (skipped=1)
2022-09-27T16:07:32.0689787Z 
2022-09-27T16:07:32.0689896Z Generating XML reports...
2022-09-27T16:07:32.0725297Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160732.xml
2022-09-27T16:07:32.6554561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38dm7157
2022-09-27T16:07:32.6555184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38dm7157/_remote_module_non_scriptable.py
2022-09-27T16:07:32.7479254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjqce8at
2022-09-27T16:07:32.7480126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjqce8at/_remote_module_non_scriptable.py
2022-09-27T16:07:32.7823472Z [1664294852.782013] [28fd6574525b:45626:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:32.7823979Z [1664294852.782013] [28fd6574525b:45626:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:32.7849100Z [1664294852.784626] [28fd6574525b:45627:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:32.7849601Z [1664294852.784626] [28fd6574525b:45627:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:33.6497392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:33.6499213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:33.6513208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:33.6517533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:34.0696398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:34.0697308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:34.0698342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:34.0699122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:34.1253169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:34.1253683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:34.1256579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:34.1257072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:34.3418767Z ok (7.035s)
2022-09-27T16:07:34.3418973Z 
2022-09-27T16:07:34.3419374Z ----------------------------------------------------------------------
2022-09-27T16:07:34.3419722Z Ran 1 test in 7.035s
2022-09-27T16:07:34.3419890Z 
2022-09-27T16:07:34.3419987Z OK
2022-09-27T16:07:34.3420124Z 
2022-09-27T16:07:34.3420259Z Generating XML reports...
2022-09-27T16:07:34.3456437Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160727.xml
2022-09-27T16:07:34.5899459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:07:34.5921886Z 
2022-09-27T16:07:34.5922292Z Running tests...
2022-09-27T16:07:34.5923139Z ----------------------------------------------------------------------
2022-09-27T16:07:34.6423986Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:34.6448998Z 
2022-09-27T16:07:34.6449338Z Running tests...
2022-09-27T16:07:34.6449788Z ----------------------------------------------------------------------
2022-09-27T16:07:34.6469617Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:07:34.6470102Z 
2022-09-27T16:07:34.6470700Z ----------------------------------------------------------------------
2022-09-27T16:07:34.6471324Z Ran 1 test in 0.002s
2022-09-27T16:07:34.6471493Z 
2022-09-27T16:07:34.6471631Z OK (skipped=1)
2022-09-27T16:07:34.6471788Z 
2022-09-27T16:07:34.6471915Z Generating XML reports...
2022-09-27T16:07:34.6507268Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160734.xml
2022-09-27T16:07:36.2899189Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:36.3093035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45882
2022-09-27T16:07:36.3099403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45883
2022-09-27T16:07:36.4868586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:36.4869106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:36.4870845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:36.4871361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:36.7218537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:36.7219100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:36.7221200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:36.7221699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:37.0066356Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:07:37.0113314Z 
2022-09-27T16:07:37.0114723Z Running tests...
2022-09-27T16:07:37.0115829Z ----------------------------------------------------------------------
2022-09-27T16:07:37.2933807Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:37.2958886Z 
2022-09-27T16:07:37.2959308Z Running tests...
2022-09-27T16:07:37.2959765Z ----------------------------------------------------------------------
2022-09-27T16:07:37.9299792Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:37.9300381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:37.9301390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:37.9301886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:37.9531980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:37.9532461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:37.9534697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:37.9535189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:38.4782434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:38.4782993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:38.4783769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:38.4784476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:38.4890630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:38.5795968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:38.6321252Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:38.6525896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45963
2022-09-27T16:07:38.6531037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45964
2022-09-27T16:07:38.9477473Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:38.9684931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45966
2022-09-27T16:07:38.9690414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45967
2022-09-27T16:07:39.9629043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu68jaweq
2022-09-27T16:07:39.9630430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu68jaweq/_remote_module_non_scriptable.py
2022-09-27T16:07:40.0179575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0c08krl
2022-09-27T16:07:40.0180927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0c08krl/_remote_module_non_scriptable.py
2022-09-27T16:07:40.3778417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:40.3778945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:40.3781345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:40.3781847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:40.3947983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:40.3948462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:40.3950755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:40.3951241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:40.5192018Z ok (5.927s)
2022-09-27T16:07:40.5192258Z 
2022-09-27T16:07:40.5192646Z ----------------------------------------------------------------------
2022-09-27T16:07:40.5192986Z Ran 1 test in 5.927s
2022-09-27T16:07:40.5193162Z 
2022-09-27T16:07:40.5193259Z OK
2022-09-27T16:07:40.5193393Z 
2022-09-27T16:07:40.5193509Z Generating XML reports...
2022-09-27T16:07:40.5229440Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160734.xml
2022-09-27T16:07:40.7210684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:40.7211189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:40.7212790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:40.7213283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:40.7991145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:40.7991640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:40.7993009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:40.7993493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:40.8997906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:40.9143825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:40.9144634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:40.9201153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:40.9208319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:41.0158663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:41.2375622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:41.3234511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:41.3235344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:41.3287200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:41.3293694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:41.3599722Z skip: Need at least 4 CUDA devices (4.348s)
2022-09-27T16:07:41.3599954Z 
2022-09-27T16:07:41.3601216Z ----------------------------------------------------------------------
2022-09-27T16:07:41.3601622Z Ran 1 test in 4.349s
2022-09-27T16:07:41.3601795Z 
2022-09-27T16:07:41.3601920Z OK (skipped=1)
2022-09-27T16:07:41.3602087Z 
2022-09-27T16:07:41.3602213Z Generating XML reports...
2022-09-27T16:07:41.3637901Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160737.xml
2022-09-27T16:07:41.4248420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:42.7321738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:42.7322280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:42.7323697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:42.7324469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:43.2700903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:07:43.2728022Z 
2022-09-27T16:07:43.2728767Z Running tests...
2022-09-27T16:07:43.2729792Z ----------------------------------------------------------------------
2022-09-27T16:07:43.6036444Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:43.6037649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:43.6038859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:43.6039808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:44.1404304Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:07:44.1427919Z 
2022-09-27T16:07:44.1428318Z Running tests...
2022-09-27T16:07:44.1429076Z ----------------------------------------------------------------------
2022-09-27T16:07:44.5812611Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... ok (7.285s)
2022-09-27T16:07:44.5812988Z 
2022-09-27T16:07:44.5813363Z ----------------------------------------------------------------------
2022-09-27T16:07:44.5813710Z Ran 1 test in 7.285s
2022-09-27T16:07:44.5813880Z 
2022-09-27T16:07:44.5813973Z OK
2022-09-27T16:07:44.5814113Z 
2022-09-27T16:07:44.5814240Z Generating XML reports...
2022-09-27T16:07:44.5848860Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160737.xml
2022-09-27T16:07:44.9742129Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:44.9946897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46186
2022-09-27T16:07:44.9952363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46187
2022-09-27T16:07:46.0204669Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:46.0404403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46283
2022-09-27T16:07:46.0410085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46284
2022-09-27T16:07:46.7345374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:46.7357490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:46.7359798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:46.7367775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:46.8212116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:46.8212650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:46.8214104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:46.8214588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:46.8535170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:46.8535663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:46.8537924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:46.8538404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:47.3122501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:47.3145022Z 
2022-09-27T16:07:47.3145352Z Running tests...
2022-09-27T16:07:47.3146042Z ----------------------------------------------------------------------
2022-09-27T16:07:47.3822752Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:47.3823633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:47.3824395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:47.3825096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:47.3831487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:47.3832277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:47.6657081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:47.6657608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:47.6659696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:47.6660561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:47.7406904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:47.7407419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:47.7408767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:47.7409254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:48.1980402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:48.2913611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:48.2914455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:48.2996741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:48.3003699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:48.3927893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:48.7557878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbw_i2u9
2022-09-27T16:07:48.7558820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbw_i2u9/_remote_module_non_scriptable.py
2022-09-27T16:07:48.7873048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36n589ru
2022-09-27T16:07:48.7874746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36n589ru/_remote_module_non_scriptable.py
2022-09-27T16:07:48.8066131Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:07:48.8066998Z   warnings.warn(
2022-09-27T16:07:48.8068222Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:07:48.8068967Z   warnings.warn(
2022-09-27T16:07:48.8478062Z skip: Need at least 4 CUDA devices (4.705s)
2022-09-27T16:07:48.8478309Z 
2022-09-27T16:07:48.8478700Z ----------------------------------------------------------------------
2022-09-27T16:07:48.8479037Z Ran 1 test in 4.705s
2022-09-27T16:07:48.8479203Z 
2022-09-27T16:07:48.8479297Z OK (skipped=1)
2022-09-27T16:07:48.8479452Z 
2022-09-27T16:07:48.8479586Z Generating XML reports...
2022-09-27T16:07:48.8514788Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160744.xml
2022-09-27T16:07:48.9956264Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:49.0126734Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.698s)
2022-09-27T16:07:49.0127338Z 
2022-09-27T16:07:49.0127617Z ----------------------------------------------------------------------
2022-09-27T16:07:49.0127952Z Ran 1 test in 1.698s
2022-09-27T16:07:49.0128406Z 
2022-09-27T16:07:49.0128523Z OK (skipped=1)
2022-09-27T16:07:49.0128682Z 
2022-09-27T16:07:49.0128794Z Generating XML reports...
2022-09-27T16:07:49.0161192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160747.xml
2022-09-27T16:07:49.2675182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:49.2675727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:49.2751775Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:49.2753503Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:07:49.2909869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:49.2910425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:49.3002625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:49.3003130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:49.8067303Z ok (6.534s)
2022-09-27T16:07:49.8067480Z 
2022-09-27T16:07:49.8067871Z ----------------------------------------------------------------------
2022-09-27T16:07:49.8068214Z Ran 1 test in 6.534s
2022-09-27T16:07:49.8068379Z 
2022-09-27T16:07:49.8068483Z OK
2022-09-27T16:07:49.8068619Z 
2022-09-27T16:07:49.8068734Z Generating XML reports...
2022-09-27T16:07:49.8123623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160743.xml
2022-09-27T16:07:51.2231705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:51.2233081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:51.2234328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:51.2235314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:51.2562446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:51.2563282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:51.2566354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:51.2567228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:51.7210566Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:07:51.7234747Z 
2022-09-27T16:07:51.7235473Z Running tests...
2022-09-27T16:07:51.7235999Z ----------------------------------------------------------------------
2022-09-27T16:07:51.7396505Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:51.7420405Z 
2022-09-27T16:07:51.7420698Z Running tests...
2022-09-27T16:07:51.7421116Z ----------------------------------------------------------------------
2022-09-27T16:07:51.9829393Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:51.9829977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:51.9831473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:51.9831961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:52.4856083Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:07:52.4877648Z 
2022-09-27T16:07:52.4878067Z Running tests...
2022-09-27T16:07:52.4878538Z ----------------------------------------------------------------------
2022-09-27T16:07:53.3548378Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:53.3612526Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:53.3750213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46472
2022-09-27T16:07:53.3757081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46473
2022-09-27T16:07:53.3821226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46475
2022-09-27T16:07:53.3828057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46476
2022-09-27T16:07:54.1180460Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:07:54.1481021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46602
2022-09-27T16:07:54.1487178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46603
2022-09-27T16:07:55.0854197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:55.0854788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:55.0855580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:55.0856074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:55.1216620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:55.1217131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:55.1219033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:55.1219812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:55.1246133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:55.1246630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:55.1249982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:55.1250480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:55.1895389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:55.1895894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:55.1897602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:55.1898108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:55.5990205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:55.6314291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:55.6355683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:55.6356693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:55.6395567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:55.6402823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:55.6406389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:07:55.7018912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:55.7019667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:55.7023357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:55.7030175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:55.7371555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:55.7374468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:07:55.7375252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:55.7420872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:07:55.8033218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:55.8338252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:55.8338769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:55.8341194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:55.8341672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:55.8857660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:55.8858152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:55.8859784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:55.8860277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:56.1897547Z ok (4.447s)
2022-09-27T16:07:56.1897763Z 
2022-09-27T16:07:56.1898162Z ----------------------------------------------------------------------
2022-09-27T16:07:56.1898500Z Ran 1 test in 4.447s
2022-09-27T16:07:56.1898658Z 
2022-09-27T16:07:56.1898754Z OK
2022-09-27T16:07:56.1898888Z 
2022-09-27T16:07:56.1899022Z Generating XML reports...
2022-09-27T16:07:56.1935470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160751.xml
2022-09-27T16:07:56.4580158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:07:56.4580733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:07:56.4581525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:56.4582234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:07:56.4589045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:07:56.4589543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:07:57.0980296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy9vxvbn
2022-09-27T16:07:57.0980945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy9vxvbn/_remote_module_non_scriptable.py
2022-09-27T16:07:57.1808771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbk4smcua
2022-09-27T16:07:57.1809897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbk4smcua/_remote_module_non_scriptable.py
2022-09-27T16:07:57.2173160Z [1664294877.216858] [28fd6574525b:46472:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:57.2173674Z [1664294877.216870] [28fd6574525b:46473:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:07:57.2174158Z [1664294877.216858] [28fd6574525b:46472:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:57.2174635Z [1664294877.216870] [28fd6574525b:46473:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:07:57.9096374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnu603oz
2022-09-27T16:07:57.9097000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnu603oz/_remote_module_non_scriptable.py
2022-09-27T16:07:57.9708804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjp_8mzg
2022-09-27T16:07:57.9710295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjp_8mzg/_remote_module_non_scriptable.py
2022-09-27T16:07:57.9910760Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:07:57.9911560Z   warnings.warn(
2022-09-27T16:07:57.9912637Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:07:57.9913362Z   warnings.warn(
2022-09-27T16:07:58.0577241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:58.0577804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:07:58.0875360Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-09-27T16:07:58.0875771Z Traceback (most recent call last):
2022-09-27T16:07:58.0876432Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:07:58.0876839Z     getattr(self, test_name)()
2022-09-27T16:07:58.0877371Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:07:58.0877733Z     fn()
2022-09-27T16:07:58.0889329Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 145, in wrapper
2022-09-27T16:07:58.0889858Z     return func(*args, **kwargs)
2022-09-27T16:07:58.0890583Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 5178, in test_post_localSGD_optimizer_step_reload
2022-09-27T16:07:58.0891124Z     self._test_post_localSGD_optimizer_step_reload(
2022-09-27T16:07:58.0892089Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 5088, in _test_post_localSGD_optimizer_step_reload
2022-09-27T16:07:58.0892655Z     checkpoint = torch.load(chkpt_file, map_location=map_location)
2022-09-27T16:07:58.0893216Z   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 764, in load
2022-09-27T16:07:58.0893691Z     return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
2022-09-27T16:07:58.0894294Z   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 971, in _legacy_load
2022-09-27T16:07:58.0894759Z     magic_number = pickle_module.load(f, **pickle_load_args)
2022-09-27T16:07:58.0895179Z _pickle.UnpicklingError: A load persistent id instruction was encountered,
2022-09-27T16:07:58.0895614Z but no persistent_load function was specified.
2022-09-27T16:07:58.0895958Z  exiting process 1 with exit code: 10
2022-09-27T16:07:58.4542231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:07:58.4543043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:07:58.4544217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:07:58.4544716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:07:58.5859945Z Process 1 terminated with exit code 10, terminating remaining processes.
2022-09-27T16:07:58.5874119Z ERROR (6.864s)
2022-09-27T16:07:58.5874690Z     test_post_localSGD_optimizer_step_reload errored - num_retries_left: 3
2022-09-27T16:07:58.5875074Z Traceback (most recent call last):
2022-09-27T16:07:58.5875635Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 501, in wrapper
2022-09-27T16:07:58.5876054Z     self._join_processes(fn)
2022-09-27T16:07:58.5876602Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 724, in _join_processes
2022-09-27T16:07:58.5877054Z     self._check_return_codes(elapsed_time)
2022-09-27T16:07:58.5877634Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 769, in _check_return_codes
2022-09-27T16:07:58.5878059Z     raise RuntimeError(error)
2022-09-27T16:07:58.5878409Z RuntimeError: Process 1 exited with error code 10 and exception:
2022-09-27T16:07:58.5878759Z Traceback (most recent call last):
2022-09-27T16:07:58.5879283Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:07:58.5879688Z     getattr(self, test_name)()
2022-09-27T16:07:58.5880996Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:07:58.5881439Z     fn()
2022-09-27T16:07:58.5881948Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 145, in wrapper
2022-09-27T16:07:58.5882361Z     return func(*args, **kwargs)
2022-09-27T16:07:58.5882957Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 5178, in test_post_localSGD_optimizer_step_reload
2022-09-27T16:07:58.5883457Z     self._test_post_localSGD_optimizer_step_reload(
2022-09-27T16:07:58.5884109Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 5088, in _test_post_localSGD_optimizer_step_reload
2022-09-27T16:07:58.5884633Z     checkpoint = torch.load(chkpt_file, map_location=map_location)
2022-09-27T16:07:58.5885145Z   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 764, in load
2022-09-27T16:07:58.5885616Z     return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
2022-09-27T16:07:58.5886187Z   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 971, in _legacy_load
2022-09-27T16:07:58.5886751Z     magic_number = pickle_module.load(f, **pickle_load_args)
2022-09-27T16:07:58.5887156Z _pickle.UnpicklingError: A load persistent id instruction was encountered,
2022-09-27T16:07:58.5887564Z but no persistent_load function was specified.
2022-09-27T16:07:58.5887777Z 
2022-09-27T16:07:58.5887797Z 
2022-09-27T16:07:58.5887816Z 
2022-09-27T16:07:58.5896712Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46744
2022-09-27T16:07:58.5902531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46745
2022-09-27T16:07:58.9858951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:07:58.9882646Z 
2022-09-27T16:07:58.9882873Z Running tests...
2022-09-27T16:07:58.9883322Z ----------------------------------------------------------------------
2022-09-27T16:08:03.1065572Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:03.1066212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:03.6670619Z ok (11.179s)
2022-09-27T16:08:03.6670840Z 
2022-09-27T16:08:03.6671447Z ----------------------------------------------------------------------
2022-09-27T16:08:03.6671922Z Ran 1 test in 11.179s
2022-09-27T16:08:03.6672103Z 
2022-09-27T16:08:03.6672181Z OK
2022-09-27T16:08:03.6672318Z 
2022-09-27T16:08:03.6672454Z Generating XML reports...
2022-09-27T16:08:03.6707837Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160752.xml
2022-09-27T16:08:03.7446377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:03.7446864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:03.7449287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:03.7449789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:03.7900538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:03.7901046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:03.7903022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:03.7903518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:04.2940139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:04.3161861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:04.3162685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:04.3245134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:04.3252279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:04.3258777Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:04.3459227Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46817
2022-09-27T16:08:04.3464743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46818
2022-09-27T16:08:04.4176867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:05.7396109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf465xw6
2022-09-27T16:08:05.7397292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf465xw6/_remote_module_non_scriptable.py
2022-09-27T16:08:05.8223096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1yyab15
2022-09-27T16:08:05.8224211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1yyab15/_remote_module_non_scriptable.py
2022-09-27T16:08:05.8473388Z [1664294885.846923] [28fd6574525b:46744:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:05.8474521Z [1664294885.846923] [28fd6574525b:46744:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:05.8560478Z [1664294885.855702] [28fd6574525b:46745:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:05.8561318Z [1664294885.855702] [28fd6574525b:46745:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:05.8997231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:05.8998234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:05.8999442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:05.9000724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:06.0568805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:06.0569847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:06.0571054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:06.0572080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:06.0829998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:06.0830967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:06.0832267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:06.0833257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:06.4035001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:08:06.4057835Z 
2022-09-27T16:08:06.4058294Z Running tests...
2022-09-27T16:08:06.4058819Z ----------------------------------------------------------------------
2022-09-27T16:08:06.5813747Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:06.6078043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:06.6079641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:06.6119110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:06.6126105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:06.6796994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:06.6799794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:06.7090908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:07.3067866Z ok (8.719s)
2022-09-27T16:08:07.3068123Z 
2022-09-27T16:08:07.3068793Z ----------------------------------------------------------------------
2022-09-27T16:08:07.3069307Z Ran 2 tests in 15.583s
2022-09-27T16:08:07.3069581Z 
2022-09-27T16:08:07.3069734Z OK (skipped=1, expected failures=1)
2022-09-27T16:08:07.3070202Z 
2022-09-27T16:08:07.3070314Z Generating XML reports...
2022-09-27T16:08:07.3109515Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160751.xml
2022-09-27T16:08:08.1318490Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:08.1535464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46944
2022-09-27T16:08:08.1540814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46945
2022-09-27T16:08:09.6272308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:09.6272836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:09.6275327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:09.6275828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:09.7584779Z ok (10.770s)
2022-09-27T16:08:09.7584997Z 
2022-09-27T16:08:09.7585685Z ----------------------------------------------------------------------
2022-09-27T16:08:09.7586066Z Ran 1 test in 10.770s
2022-09-27T16:08:09.7586222Z 
2022-09-27T16:08:09.7586325Z OK
2022-09-27T16:08:09.7586462Z 
2022-09-27T16:08:09.7586597Z Generating XML reports...
2022-09-27T16:08:09.7620884Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160758.xml
2022-09-27T16:08:09.9028947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:09.9029471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:09.9031531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:09.9032022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:09.9160737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:09.9161283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:09.9162123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:09.9162612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:10.1259286Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:08:10.1282156Z 
2022-09-27T16:08:10.1282502Z Running tests...
2022-09-27T16:08:10.1283021Z ----------------------------------------------------------------------
2022-09-27T16:08:10.4441715Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:10.4442396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:10.4443195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:10.4443902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:10.4550165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:10.5455758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:11.8303517Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:11.8386590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpux56c4z7
2022-09-27T16:08:11.8389029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpux56c4z7/_remote_module_non_scriptable.py
2022-09-27T16:08:11.8501904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47091
2022-09-27T16:08:11.8508113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47092
2022-09-27T16:08:11.8655225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:11.8655704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:11.8659007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:11.8659533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:11.8799726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpql7suo50
2022-09-27T16:08:11.8801723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpql7suo50/_remote_module_non_scriptable.py
2022-09-27T16:08:12.3130916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:12.3131489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:12.3243173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:12.3243823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:12.3849484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:12.3873670Z 
2022-09-27T16:08:12.3873796Z Running tests...
2022-09-27T16:08:12.3874564Z ----------------------------------------------------------------------
2022-09-27T16:08:12.9653855Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... ok (6.559s)
2022-09-27T16:08:12.9654319Z 
2022-09-27T16:08:12.9654756Z ----------------------------------------------------------------------
2022-09-27T16:08:12.9655144Z Ran 1 test in 6.559s
2022-09-27T16:08:12.9655313Z 
2022-09-27T16:08:12.9655407Z OK
2022-09-27T16:08:12.9655544Z 
2022-09-27T16:08:12.9655655Z Generating XML reports...
2022-09-27T16:08:12.9689393Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160806.xml
2022-09-27T16:08:13.5567676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:13.5568175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:13.5569482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:13.5569981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:13.5641811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:13.5642570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:13.5646100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:13.5646600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:14.0906702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:14.0908204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:14.0909962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:14.0911922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:14.0957964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:14.0963827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:08:14.1003064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:14.1011240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:08:14.1013780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:14.1072355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:14.1143595Z STAGE:2022-09-27 16:08:14 47091:47091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:14.1180986Z STAGE:2022-09-27 16:08:14 47092:47092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:14.1709664Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:14.1980989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47204
2022-09-27T16:08:14.2003320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47205
2022-09-27T16:08:15.1663545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:15.1664135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:15.1665000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:15.1665478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:15.6842218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:08:15.6864578Z 
2022-09-27T16:08:15.6864874Z Running tests...
2022-09-27T16:08:15.6865322Z ----------------------------------------------------------------------
2022-09-27T16:08:15.8493966Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... [1664294895.848867] [28fd6574525b:47091:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:15.8494657Z [1664294895.848867] [28fd6574525b:47091:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:15.8700403Z [1664294895.869552] [28fd6574525b:47092:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:15.8700925Z [1664294895.869552] [28fd6574525b:47092:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:15.8923647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:15.8924123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:15.8926820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:15.8927542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:15.9098649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:15.9099134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:15.9101981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:15.9102476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:16.2637526Z STAGE:2022-09-27 16:08:16 47092:47092 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:08:16 47091:47091 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:16.2637987Z 
2022-09-27T16:08:16.4279797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:16.4323171Z STAGE:2022-09-27 16:08:16 47091:47091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:16.4330611Z STAGE:2022-09-27 16:08:16 47092:47092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:16.4334451Z STAGE:2022-09-27 16:08:16 47091:47091 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:16.4340841Z STAGE:2022-09-27 16:08:16 47092:47092 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:16.4359015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:16.4360110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:16.4380625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:16.4387200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:16.5373485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:17.1073140Z ok (4.720s)
2022-09-27T16:08:17.1073359Z 
2022-09-27T16:08:17.1073761Z ----------------------------------------------------------------------
2022-09-27T16:08:17.1074109Z Ran 1 test in 4.720s
2022-09-27T16:08:17.1074286Z 
2022-09-27T16:08:17.1074385Z OK
2022-09-27T16:08:17.1074523Z 
2022-09-27T16:08:17.1074663Z Generating XML reports...
2022-09-27T16:08:17.1109639Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160812.xml
2022-09-27T16:08:17.2583808Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:17.2788850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47286
2022-09-27T16:08:17.2794057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47287
2022-09-27T16:08:17.5642119Z ok (7.436s)
2022-09-27T16:08:17.5642342Z 
2022-09-27T16:08:17.5642751Z ----------------------------------------------------------------------
2022-09-27T16:08:17.5643095Z Ran 1 test in 7.436s
2022-09-27T16:08:17.5643268Z 
2022-09-27T16:08:17.5643347Z OK
2022-09-27T16:08:17.5643499Z 
2022-09-27T16:08:17.5643638Z Generating XML reports...
2022-09-27T16:08:17.5679742Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160810.xml
2022-09-27T16:08:19.0073103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:19.0073668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:19.0074831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:19.0075323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:19.0365870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:19.0366404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:19.0369253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:19.0369762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:19.3607590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:19.3608105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:19.3610707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:19.3611204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:19.5660922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:19.5661665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:19.5662426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:19.5663372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:19.5668936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:19.5669437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:19.7891707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:19.7892572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:19.7894038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:19.7894545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:19.8518946Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:19.8542464Z 
2022-09-27T16:08:19.8542796Z Running tests...
2022-09-27T16:08:19.8543274Z ----------------------------------------------------------------------
2022-09-27T16:08:20.3108936Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:08:20.3132642Z 
2022-09-27T16:08:20.3133045Z Running tests...
2022-09-27T16:08:20.3133590Z ----------------------------------------------------------------------
2022-09-27T16:08:20.9874439Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36erf01h
2022-09-27T16:08:20.9875333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36erf01h/_remote_module_non_scriptable.py
2022-09-27T16:08:21.0178398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfedecpc8
2022-09-27T16:08:21.0179608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfedecpc8/_remote_module_non_scriptable.py
2022-09-27T16:08:21.3173691Z STAGE:2022-09-27 16:08:21 47287:47287 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:21.3177861Z STAGE:2022-09-27 16:08:21 47286:47286 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:21.3271041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:21.3271788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:21.3485549Z STAGE:2022-09-27 16:08:21 47286:47286 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:08:21 47287:47287 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:21.3486001Z 
2022-09-27T16:08:21.4143455Z STAGE:2022-09-27 16:08:21 47286:47286 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:21.4728921Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:21.4930502Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47439
2022-09-27T16:08:21.4935399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47440
2022-09-27T16:08:21.9186174Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:21.9383087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47442
2022-09-27T16:08:21.9388661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47443
2022-09-27T16:08:22.3537499Z STAGE:2022-09-27 16:08:22 47286:47286 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:23.0918797Z ok (7.405s)
2022-09-27T16:08:23.0919020Z 
2022-09-27T16:08:23.0919409Z ----------------------------------------------------------------------
2022-09-27T16:08:23.0919752Z Ran 1 test in 7.405s
2022-09-27T16:08:23.0919923Z 
2022-09-27T16:08:23.0920024Z OK
2022-09-27T16:08:23.0920340Z 
2022-09-27T16:08:23.0920753Z Generating XML reports...
2022-09-27T16:08:23.0955598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160815.xml
2022-09-27T16:08:23.2617717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:23.2618229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:23.2619780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:23.2620254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:23.2996637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:23.2997159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:23.2998654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:23.2999139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:23.6698152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:23.6698678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:23.6699763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:23.6700355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:23.7218245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:23.7218760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:23.7221083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:23.7221858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:23.7831933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:23.8164458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:23.8165277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:23.8237928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:23.8245154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:23.8250164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:08:23.9176335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:23.9179672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:08:23.9180675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:23.9263999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:24.1798709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:24.2394947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:24.2397784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:24.2407974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:24.2416647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:24.2419187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:08:24.3408084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:24.3410056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:08:24.3411453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:24.3438960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:24.3479267Z STAGE:2022-09-27 16:08:24 47442:47442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:24.3498613Z STAGE:2022-09-27 16:08:24 47443:47443 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:24.4005719Z ok (4.546s)
2022-09-27T16:08:24.4005948Z 
2022-09-27T16:08:24.4006322Z ----------------------------------------------------------------------
2022-09-27T16:08:24.4006672Z Ran 1 test in 4.546s
2022-09-27T16:08:24.4006841Z 
2022-09-27T16:08:24.4006940Z OK
2022-09-27T16:08:24.4007078Z 
2022-09-27T16:08:24.4007202Z Generating XML reports...
2022-09-27T16:08:24.4041928Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160819.xml
2022-09-27T16:08:25.1972657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:25.1973167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:25.1974391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:25.1974894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:25.7810962Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:08:25.7835881Z 
2022-09-27T16:08:25.7836168Z Running tests...
2022-09-27T16:08:25.7836632Z ----------------------------------------------------------------------
2022-09-27T16:08:26.0481808Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... [1664294906.047649] [28fd6574525b:47442:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:26.0482440Z [1664294906.047649] [28fd6574525b:47442:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:26.0694408Z [1664294906.068964] [28fd6574525b:47443:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:26.0694938Z [1664294906.068964] [28fd6574525b:47443:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:26.4482514Z STAGE:2022-09-27 16:08:26 47443:47443 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:08:26 47442:47442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:26.4482970Z 
2022-09-27T16:08:26.5273191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:26.5273698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:26.5276157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:26.5276652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:26.5706518Z STAGE:2022-09-27 16:08:26 47443:47443 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:26.5778559Z STAGE:2022-09-27 16:08:26 47442:47442 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:26.5789054Z STAGE:2022-09-27 16:08:26 47443:47443 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:27.0434318Z STAGE:2022-09-27 16:08:26 47442:47442 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:27.0434965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:27.0456664Z 
2022-09-27T16:08:27.0456805Z Running tests...
2022-09-27T16:08:27.0457840Z ----------------------------------------------------------------------
2022-09-27T16:08:27.3984564Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:27.4187515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47666
2022-09-27T16:08:27.4193095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47667
2022-09-27T16:08:27.8531217Z ok (7.539s)
2022-09-27T16:08:27.8531621Z 
2022-09-27T16:08:27.8532293Z ----------------------------------------------------------------------
2022-09-27T16:08:27.8532958Z Ran 1 test in 7.540s
2022-09-27T16:08:27.8533273Z 
2022-09-27T16:08:27.8533430Z OK
2022-09-27T16:08:27.8533701Z 
2022-09-27T16:08:27.8533959Z Generating XML reports...
2022-09-27T16:08:27.8568557Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160820.xml
2022-09-27T16:08:28.7424688Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:28.7618190Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47732
2022-09-27T16:08:28.7623895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47733
2022-09-27T16:08:29.1091835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:29.1092808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:29.1093985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:29.1094530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:29.1271086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:29.1271566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:29.1274769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:29.1275267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:29.7346293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:29.7347637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:29.7349138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:29.7350509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:29.7355836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:29.7356782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:29.9516761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:29.9517273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:29.9519761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:29.9520454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:30.4545229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:30.4546094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:30.4547567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:30.4548039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:30.4644457Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:08:30.4668812Z 
2022-09-27T16:08:30.4669025Z Running tests...
2022-09-27T16:08:30.4669470Z ----------------------------------------------------------------------
2022-09-27T16:08:30.4950485Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:30.4951059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:30.4953649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:30.4954156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:30.9771738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:31.0420451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:31.0421283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:31.0483391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:31.0490561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:31.0494400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:08:31.1219648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslgks69d
2022-09-27T16:08:31.1220722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslgks69d/_remote_module_non_scriptable.py
2022-09-27T16:08:31.1432944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:31.1435669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:08:31.1436442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:31.1510705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:31.1537114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p9dvyn4
2022-09-27T16:08:31.1539765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p9dvyn4/_remote_module_non_scriptable.py
2022-09-27T16:08:31.6701331Z ok (4.624s)
2022-09-27T16:08:31.6701647Z 
2022-09-27T16:08:31.6702249Z ----------------------------------------------------------------------
2022-09-27T16:08:31.6702582Z Ran 1 test in 4.624s
2022-09-27T16:08:31.6702756Z 
2022-09-27T16:08:31.6702862Z OK
2022-09-27T16:08:31.6702999Z 
2022-09-27T16:08:31.6703134Z Generating XML reports...
2022-09-27T16:08:31.6753299Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160827.xml
2022-09-27T16:08:31.7304098Z ok (5.946s)
2022-09-27T16:08:31.7304434Z 
2022-09-27T16:08:31.7305180Z ----------------------------------------------------------------------
2022-09-27T16:08:31.7305829Z Ran 1 test in 5.947s
2022-09-27T16:08:31.7306011Z 
2022-09-27T16:08:31.7306107Z OK
2022-09-27T16:08:31.7306266Z 
2022-09-27T16:08:31.7306401Z Generating XML reports...
2022-09-27T16:08:31.7350333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160825.xml
2022-09-27T16:08:32.1019520Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:32.1255853Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47854
2022-09-27T16:08:32.1261377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47855
2022-09-27T16:08:33.9119764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:33.9120570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:33.9121604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:33.9122093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:33.9172319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:33.9172769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:33.9177261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:33.9177751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:33.9446480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:33.9446932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:33.9450121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:33.9450609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:33.9632267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:33.9632720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:33.9636791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:33.9637291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:34.4429182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:34.4451747Z 
2022-09-27T16:08:34.4451886Z Running tests...
2022-09-27T16:08:34.4452631Z ----------------------------------------------------------------------
2022-09-27T16:08:34.4469008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:34.4595701Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:34.4596548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:34.4616367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:08:34.4641497Z 
2022-09-27T16:08:34.4641771Z Running tests...
2022-09-27T16:08:34.4642195Z ----------------------------------------------------------------------
2022-09-27T16:08:34.4673360Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:34.4680829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:34.4684459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:08:34.5611107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:34.5613740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:08:34.5614542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:34.5699192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:34.5744952Z STAGE:2022-09-27 16:08:34 47855:47855 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:34.5822215Z STAGE:2022-09-27 16:08:34 47854:47854 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:36.1684773Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:36.1886202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47999
2022-09-27T16:08:36.1890848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48000
2022-09-27T16:08:36.2115306Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:36.2324620Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48002
2022-09-27T16:08:36.2329919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48003
2022-09-27T16:08:36.2546277Z [1664294916.254063] [28fd6574525b:47854:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:36.2546804Z [1664294916.254063] [28fd6574525b:47854:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:36.2642559Z [1664294916.263852] [28fd6574525b:47855:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:36.2643032Z [1664294916.263852] [28fd6574525b:47855:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:36.6374855Z STAGE:2022-09-27 16:08:36 47854:47854 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:08:36 47855:47855 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:36.6375291Z 
2022-09-27T16:08:36.7693146Z STAGE:2022-09-27 16:08:36 47854:47854 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:36.7701964Z STAGE:2022-09-27 16:08:36 47854:47854 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:36.7762240Z STAGE:2022-09-27 16:08:36 47855:47855 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:36.7772410Z STAGE:2022-09-27 16:08:36 47855:47855 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:37.8914995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:37.8915509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:37.8917112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:37.8917632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:37.9224814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:37.9225310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:37.9227799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:37.9228292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:37.9426656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:37.9427105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:37.9431088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:37.9431585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:37.9757675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:37.9758414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:37.9761901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:37.9762390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:38.0387586Z ok (7.571s)
2022-09-27T16:08:38.0388088Z 
2022-09-27T16:08:38.0388861Z ----------------------------------------------------------------------
2022-09-27T16:08:38.0389572Z Ran 1 test in 7.572s
2022-09-27T16:08:38.0389890Z 
2022-09-27T16:08:38.0390084Z OK
2022-09-27T16:08:38.0390326Z 
2022-09-27T16:08:38.0390583Z Generating XML reports...
2022-09-27T16:08:38.0425779Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160830.xml
2022-09-27T16:08:38.4390299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:38.5250203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:38.5251255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:38.5300830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:38.5308078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:38.5341002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:38.5341503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:38.5342212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:38.5342909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:38.5452500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:38.6263391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:38.6355807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:39.9987261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp937_2aso
2022-09-27T16:08:39.9987877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp937_2aso/_remote_module_non_scriptable.py
2022-09-27T16:08:40.0107112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxow2tca
2022-09-27T16:08:40.0108618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxow2tca/_remote_module_non_scriptable.py
2022-09-27T16:08:40.0118495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeobp0rfb
2022-09-27T16:08:40.0121286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeobp0rfb/_remote_module_non_scriptable.py
2022-09-27T16:08:40.0306563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68_lk1ls
2022-09-27T16:08:40.0308512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68_lk1ls/_remote_module_non_scriptable.py
2022-09-27T16:08:40.1968468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:40.1969025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:40.1970383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:40.1970904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:40.6438945Z ok (6.179s)
2022-09-27T16:08:40.6439205Z 
2022-09-27T16:08:40.6439921Z ----------------------------------------------------------------------
2022-09-27T16:08:40.6440535Z Ran 1 test in 6.180s
2022-09-27T16:08:40.6440705Z 
2022-09-27T16:08:40.6440800Z OK
2022-09-27T16:08:40.6440921Z 
2022-09-27T16:08:40.6441056Z Generating XML reports...
2022-09-27T16:08:40.6491399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160834.xml
2022-09-27T16:08:40.7425044Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:08:40.7447807Z 
2022-09-27T16:08:40.7448040Z Running tests...
2022-09-27T16:08:40.7448494Z ----------------------------------------------------------------------
2022-09-27T16:08:41.1004478Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... ok (6.655s)
2022-09-27T16:08:41.1004838Z 
2022-09-27T16:08:41.1005219Z ----------------------------------------------------------------------
2022-09-27T16:08:41.1005562Z Ran 1 test in 6.655s
2022-09-27T16:08:41.1005739Z 
2022-09-27T16:08:41.1005833Z OK
2022-09-27T16:08:41.1005957Z 
2022-09-27T16:08:41.1006084Z Generating XML reports...
2022-09-27T16:08:41.1056209Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160834.xml
2022-09-27T16:08:42.5246610Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:42.5445549Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48265
2022-09-27T16:08:42.5452487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48266
2022-09-27T16:08:42.9076262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:42.9076755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:42.9078723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:42.9079227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:43.3656242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:43.3656758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:43.3659778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:43.3660271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:43.4582981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:08:43.4606814Z 
2022-09-27T16:08:43.4606959Z Running tests...
2022-09-27T16:08:43.4607707Z ----------------------------------------------------------------------
2022-09-27T16:08:43.8742851Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:43.8766030Z 
2022-09-27T16:08:43.8766774Z Running tests...
2022-09-27T16:08:43.8767268Z ----------------------------------------------------------------------
2022-09-27T16:08:43.8790682Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-09-27T16:08:43.8791139Z 
2022-09-27T16:08:43.8791519Z ----------------------------------------------------------------------
2022-09-27T16:08:43.8791859Z Ran 1 test in 0.003s
2022-09-27T16:08:43.8792031Z 
2022-09-27T16:08:43.8792144Z OK (skipped=1)
2022-09-27T16:08:43.8792304Z 
2022-09-27T16:08:43.8792433Z Generating XML reports...
2022-09-27T16:08:43.8827976Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160843.xml
2022-09-27T16:08:44.1591116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:44.1591653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:44.1592576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:44.1593060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:44.1979413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:44.1979936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:44.1980857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:44.1981349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:44.6765162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:44.7068778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:44.7069582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:44.7070294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:44.7077432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:44.7077928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:44.7081117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:08:44.7081641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:08:44.7082424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:44.7083454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:44.7178850Z STAGE:2022-09-27 16:08:44 48266:48266 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:44.7179447Z STAGE:2022-09-27 16:08:44 48265:48265 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:45.0962561Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:45.1170678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48377
2022-09-27T16:08:45.1175794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48378
2022-09-27T16:08:45.9588734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:45.9589545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:45.9590492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:45.9591003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:46.3863133Z [1664294926.385795] [28fd6574525b:48266:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:46.3863696Z [1664294926.385795] [28fd6574525b:48266:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:46.4257662Z [1664294926.425265] [28fd6574525b:48265:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:08:46.4258177Z [1664294926.425265] [28fd6574525b:48265:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:08:46.4723670Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:46.4745806Z 
2022-09-27T16:08:46.4745956Z Running tests...
2022-09-27T16:08:46.4746895Z ----------------------------------------------------------------------
2022-09-27T16:08:46.4764027Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-09-27T16:08:46.4764329Z 
2022-09-27T16:08:46.4764627Z ----------------------------------------------------------------------
2022-09-27T16:08:46.4764962Z Ran 1 test in 0.002s
2022-09-27T16:08:46.4765127Z 
2022-09-27T16:08:46.4765382Z OK (skipped=1)
2022-09-27T16:08:46.4765556Z 
2022-09-27T16:08:46.4765687Z Generating XML reports...
2022-09-27T16:08:46.4800160Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160846.xml
2022-09-27T16:08:46.7954774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:46.7955335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:46.7956804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:46.7957306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:46.8381331Z STAGE:2022-09-27 16:08:46 48265:48265 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:08:46 48266:48266 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:46.8381782Z 
2022-09-27T16:08:46.8446287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:46.8446773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:46.8451068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:46.8451561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:46.9879215Z STAGE:2022-09-27 16:08:46 48265:48265 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:46.9879865Z STAGE:2022-09-27 16:08:46 48266:48266 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:08:46.9889433Z STAGE:2022-09-27 16:08:46 48265:48265 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:46.9890113Z STAGE:2022-09-27 16:08:46 48266:48266 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:08:47.3841596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:47.3842233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:47.3843258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:47.3844317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:47.3949477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:47.4933295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:48.1572923Z ok (7.412s)
2022-09-27T16:08:48.1573136Z 
2022-09-27T16:08:48.1573544Z ----------------------------------------------------------------------
2022-09-27T16:08:48.1573902Z Ran 1 test in 7.412s
2022-09-27T16:08:48.1574052Z 
2022-09-27T16:08:48.1574152Z OK
2022-09-27T16:08:48.1574289Z 
2022-09-27T16:08:48.1574429Z Generating XML reports...
2022-09-27T16:08:48.1610420Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160840.xml
2022-09-27T16:08:48.4646552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:48.4647145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:48.4648667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:48.4649718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:48.8350112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmt4kenzt
2022-09-27T16:08:48.8350757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmt4kenzt/_remote_module_non_scriptable.py
2022-09-27T16:08:48.9051793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod480xd6
2022-09-27T16:08:48.9052511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod480xd6/_remote_module_non_scriptable.py
2022-09-27T16:08:49.0036828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:49.0060647Z 
2022-09-27T16:08:49.0061108Z Running tests...
2022-09-27T16:08:49.0061642Z ----------------------------------------------------------------------
2022-09-27T16:08:49.0080070Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-09-27T16:08:49.0080676Z 
2022-09-27T16:08:49.0080974Z ----------------------------------------------------------------------
2022-09-27T16:08:49.0081565Z Ran 1 test in 0.002s
2022-09-27T16:08:49.0081751Z 
2022-09-27T16:08:49.0081845Z OK (skipped=1)
2022-09-27T16:08:49.0082004Z 
2022-09-27T16:08:49.0082133Z Generating XML reports...
2022-09-27T16:08:49.0117046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160848.xml
2022-09-27T16:08:49.4255728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:49.4257653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:08:49.9281083Z ok (6.467s)
2022-09-27T16:08:49.9281367Z 
2022-09-27T16:08:49.9281785Z ----------------------------------------------------------------------
2022-09-27T16:08:49.9282107Z Ran 1 test in 6.467s
2022-09-27T16:08:49.9282273Z 
2022-09-27T16:08:49.9282369Z OK
2022-09-27T16:08:49.9282520Z 
2022-09-27T16:08:49.9282655Z Generating XML reports...
2022-09-27T16:08:49.9317001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160843.xml
2022-09-27T16:08:50.4099681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:50.4100411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:50.4101844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:50.4102321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:50.8961620Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:08:50.8984167Z 
2022-09-27T16:08:50.8984424Z Running tests...
2022-09-27T16:08:50.8984875Z ----------------------------------------------------------------------
2022-09-27T16:08:51.0804639Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:51.0805911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:51.0808695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:51.0809781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:51.6041954Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:51.6065733Z 
2022-09-27T16:08:51.6065908Z Running tests...
2022-09-27T16:08:51.6066374Z ----------------------------------------------------------------------
2022-09-27T16:08:51.6084783Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-09-27T16:08:51.6085293Z 
2022-09-27T16:08:51.6086028Z ----------------------------------------------------------------------
2022-09-27T16:08:51.6086401Z Ran 1 test in 0.002s
2022-09-27T16:08:51.6086572Z 
2022-09-27T16:08:51.6086695Z OK (skipped=1)
2022-09-27T16:08:51.6086852Z 
2022-09-27T16:08:51.6086963Z Generating XML reports...
2022-09-27T16:08:51.6120918Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160851.xml
2022-09-27T16:08:52.0192437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:52.0192949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:52.0194200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:52.0194695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:52.5153583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:08:52.5175750Z 
2022-09-27T16:08:52.5176005Z Running tests...
2022-09-27T16:08:52.5176439Z ----------------------------------------------------------------------
2022-09-27T16:08:52.5236439Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:52.5430657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48598
2022-09-27T16:08:52.5436785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48599
2022-09-27T16:08:53.6144777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:53.6145300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:53.6146337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:53.6146812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:54.1152462Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:54.1216791Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:08:54.1241048Z 
2022-09-27T16:08:54.1241467Z Running tests...
2022-09-27T16:08:54.1241949Z ----------------------------------------------------------------------
2022-09-27T16:08:54.1317577Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2022-09-27T16:08:54.1318654Z 
2022-09-27T16:08:54.1318938Z ----------------------------------------------------------------------
2022-09-27T16:08:54.1319280Z Ran 1 test in 1.614s
2022-09-27T16:08:54.1319445Z 
2022-09-27T16:08:54.1319562Z OK (skipped=1)
2022-09-27T16:08:54.1319717Z 
2022-09-27T16:08:54.1319840Z Generating XML reports...
2022-09-27T16:08:54.1351266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160852.xml
2022-09-27T16:08:54.2077037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:54.2077519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:54.2078095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:54.2078554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:54.2079749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:54.2080471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:54.2081367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:54.2081845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:54.7518806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:54.7520812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:54.7521753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:54.7619582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:54.7626273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:54.8536161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:55.2527056Z skip: Skipped due to small world size. (4.352s)
2022-09-27T16:08:55.2527837Z 
2022-09-27T16:08:55.2528840Z ----------------------------------------------------------------------
2022-09-27T16:08:55.2529990Z Ran 1 test in 4.353s
2022-09-27T16:08:55.2530717Z 
2022-09-27T16:08:55.2530981Z OK (skipped=1)
2022-09-27T16:08:55.2531338Z 
2022-09-27T16:08:55.2531991Z Generating XML reports...
2022-09-27T16:08:55.2602486Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160850.xml
2022-09-27T16:08:55.8537929Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:55.8739977Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48734
2022-09-27T16:08:55.8746084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48735
2022-09-27T16:08:56.2672470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:56.2672990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:56.2675389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:56.2675888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:56.9148711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:08:56.9171342Z 
2022-09-27T16:08:56.9171658Z Running tests...
2022-09-27T16:08:56.9172100Z ----------------------------------------------------------------------
2022-09-27T16:08:57.5239952Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:57.5240925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:57.5241787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:57.5242293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:57.6009254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:57.6009750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:57.6011462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:57.6011952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:57.6203531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:08:57.6204039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:08:57.6206239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:08:57.6206998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:08:58.0342140Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:08:58.0366240Z 
2022-09-27T16:08:58.0366654Z Running tests...
2022-09-27T16:08:58.0367112Z ----------------------------------------------------------------------
2022-09-27T16:08:58.1325356Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:08:58.1468854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:08:58.1469681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:58.1528950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:08:58.1535811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:08:58.1539629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:08:58.2482813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:08:58.2485955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:08:58.2486747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:58.2554706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:08:58.2665084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:08:58.2665611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:08:58.2666303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:08:58.2666981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:08:58.5117693Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:58.5292933Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-09-27T16:08:58.5293636Z 
2022-09-27T16:08:58.5294223Z ----------------------------------------------------------------------
2022-09-27T16:08:58.5294581Z Ran 1 test in 1.612s
2022-09-27T16:08:58.5294745Z 
2022-09-27T16:08:58.5294855Z OK (skipped=1)
2022-09-27T16:08:58.5295026Z 
2022-09-27T16:08:58.5295135Z Generating XML reports...
2022-09-27T16:08:58.5328080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160856.xml
2022-09-27T16:08:59.7527143Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:08:59.7824044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48887
2022-09-27T16:08:59.7911862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48888
2022-09-27T16:09:00.7331766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:00.7332403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:00.7333934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:00.7334409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:00.9195530Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:09:01.2283441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:09:01.2306694Z 
2022-09-27T16:09:01.2306983Z Running tests...
2022-09-27T16:09:01.2307418Z ----------------------------------------------------------------------
2022-09-27T16:09:01.4745738Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:01.4746352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:01.4747385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:01.4747883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:01.4852446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:01.4852946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:01.4856946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:01.4857425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:01.9942856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:02.0186581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:02.0187458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:02.0247260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:02.0254887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:02.1201103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:02.4979589Z skip: Skipped due to small world size. (4.461s)
2022-09-27T16:09:02.4979945Z 
2022-09-27T16:09:02.4980628Z ----------------------------------------------------------------------
2022-09-27T16:09:02.4981352Z Ran 1 test in 4.461s
2022-09-27T16:09:02.4981611Z 
2022-09-27T16:09:02.4981722Z OK (skipped=1)
2022-09-27T16:09:02.4981877Z 
2022-09-27T16:09:02.4982011Z Generating XML reports...
2022-09-27T16:09:02.5017474Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160858.xml
2022-09-27T16:09:02.8069496Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:02.8266573Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48963
2022-09-27T16:09:02.8272767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48964
2022-09-27T16:09:04.5508796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:04.5509337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:04.5511599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:04.5512095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:04.5890922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:04.5891414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:04.5895265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:04.5895756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:04.6512939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:04.6513470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:04.6515024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:04.6515509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:05.1117449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:05.1118019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:05.1118835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:05.1119527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:05.1225567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:05.1308999Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:09:05.1332678Z 
2022-09-27T16:09:05.1333076Z Running tests...
2022-09-27T16:09:05.1333584Z ----------------------------------------------------------------------
2022-09-27T16:09:05.2132071Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:06.5204007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx_i93ds
2022-09-27T16:09:06.5204787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx_i93ds/_remote_module_non_scriptable.py
2022-09-27T16:09:06.7702988Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:06.7897728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49074
2022-09-27T16:09:06.7903765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49075
2022-09-27T16:09:08.3831739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:08.3832246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:08.3833755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:08.3834246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:08.4669551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:08.4670305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:08.4673190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:08.4673694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:08.8978816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:08.9700596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:08.9701423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:08.9789340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:08.9796030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:09.0715178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:09.3966953Z skip: Skipped due to small world size. (4.263s)
2022-09-27T16:09:09.3967743Z 
2022-09-27T16:09:09.3968382Z ----------------------------------------------------------------------
2022-09-27T16:09:09.3969032Z Ran 1 test in 4.263s
2022-09-27T16:09:09.3969325Z 
2022-09-27T16:09:09.3969527Z OK (skipped=1)
2022-09-27T16:09:09.3969813Z 
2022-09-27T16:09:09.3970050Z Generating XML reports...
2022-09-27T16:09:09.4006347Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160905.xml
2022-09-27T16:09:11.3767628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:11.3768128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:11.3771051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:11.3771559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:11.8863285Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:09:11.8887497Z 
2022-09-27T16:09:11.8887854Z Running tests...
2022-09-27T16:09:11.8888310Z ----------------------------------------------------------------------
2022-09-27T16:09:13.4673018Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:13.4869809Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49177
2022-09-27T16:09:13.4877406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49178
2022-09-27T16:09:15.0684174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:15.0685331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:15.0686728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:15.0687894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:15.0941050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:15.0942182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:15.0944592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:15.0945757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:15.5824351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:15.6041232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:15.6043510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:15.6128969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:15.6135261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:15.7055580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:15.9940219Z skip: Skipped due to small world size. (4.105s)
2022-09-27T16:09:15.9940812Z 
2022-09-27T16:09:15.9941530Z ----------------------------------------------------------------------
2022-09-27T16:09:15.9942204Z Ran 1 test in 4.105s
2022-09-27T16:09:15.9942545Z 
2022-09-27T16:09:15.9942767Z OK (skipped=1)
2022-09-27T16:09:15.9943046Z 
2022-09-27T16:09:15.9943316Z Generating XML reports...
2022-09-27T16:09:15.9977858Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160911.xml
2022-09-27T16:09:16.5156172Z ok (22.391s)
2022-09-27T16:09:16.5156553Z 
2022-09-27T16:09:16.5157107Z ----------------------------------------------------------------------
2022-09-27T16:09:16.5157756Z Ran 1 test in 22.391s
2022-09-27T16:09:16.5157910Z 
2022-09-27T16:09:16.5158007Z OK
2022-09-27T16:09:16.5158143Z 
2022-09-27T16:09:16.5158279Z Generating XML reports...
2022-09-27T16:09:16.5193834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160854.xml
2022-09-27T16:09:17.2206866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfl2avp4
2022-09-27T16:09:17.2208692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfl2avp4/_remote_module_non_scriptable.py
2022-09-27T16:09:18.2577822Z ok (17.027s)
2022-09-27T16:09:18.2578040Z 
2022-09-27T16:09:18.2578435Z ----------------------------------------------------------------------
2022-09-27T16:09:18.2578801Z Ran 1 test in 17.027s
2022-09-27T16:09:18.2578955Z 
2022-09-27T16:09:18.2579056Z OK
2022-09-27T16:09:18.2579192Z 
2022-09-27T16:09:18.2579327Z Generating XML reports...
2022-09-27T16:09:18.2615173Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160901.xml
2022-09-27T16:09:18.2918311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:18.2918827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:18.2920539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:18.2921019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:18.6537842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:18.6538344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:18.6540418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:18.6540923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:18.7986807Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:09:18.8010722Z 
2022-09-27T16:09:18.8011136Z Running tests...
2022-09-27T16:09:18.8011593Z ----------------------------------------------------------------------
2022-09-27T16:09:19.1602812Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:09:19.1624948Z 
2022-09-27T16:09:19.1625115Z Running tests...
2022-09-27T16:09:19.1626130Z ----------------------------------------------------------------------
2022-09-27T16:09:20.3087418Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:20.3088074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:20.3089295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:20.3089783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:20.4930009Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:20.5128920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49351
2022-09-27T16:09:20.5134669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49352
2022-09-27T16:09:20.7753531Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:20.7947192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49354
2022-09-27T16:09:20.7953536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49355
2022-09-27T16:09:20.8060258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:09:20.8083871Z 
2022-09-27T16:09:20.8084426Z Running tests...
2022-09-27T16:09:20.8084868Z ----------------------------------------------------------------------
2022-09-27T16:09:22.2337171Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:22.2338261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:22.2339108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:22.2339602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.2364392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:22.2364871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:22.2368271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:22.2368782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.4582692Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:22.4782848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49484
2022-09-27T16:09:22.4788292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49485
2022-09-27T16:09:22.5304998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:22.5305475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:22.5307629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:22.5308152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.5328921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:22.5329417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:22.5332827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:22.5333323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:22.7515989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:22.7666699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:22.7667499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:22.7719444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:22.7726119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:22.7792677Z STAGE:2022-09-27 16:09:22 49351:49351 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:22.8680115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:22.8746518Z STAGE:2022-09-27 16:09:22 49352:49352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:23.0439981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:23.0500141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:23.0500971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:23.0541977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:23.0549047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:23.0552804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:09:23.1550480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:23.1552079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:09:23.1554561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:09:23.1577446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:09:23.1713054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:09:23.1714648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:09:23.1716734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:09:23.1718503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:09:24.1878732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:24.1879246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:24.1881028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:24.1881528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:24.2207076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:24.2207596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:24.2209373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:24.2209845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:24.6750296Z [1664294964.674516] [28fd6574525b:49351:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:09:24.6750845Z [1664294964.674516] [28fd6574525b:49351:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:09:24.7108403Z [1664294964.710407] [28fd6574525b:49352:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:09:24.7109098Z [1664294964.710407] [28fd6574525b:49352:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:09:24.7711982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:24.7712847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:24.7713399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:24.7714063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:24.7821132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:24.8727274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:24.8835996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:09:24.8836678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:09:24.8837493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:09:24.8838444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:09:25.1134010Z STAGE:2022-09-27 16:09:25 49352:49352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:25.1151423Z STAGE:2022-09-27 16:09:25 49351:49351 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:25.2418644Z STAGE:2022-09-27 16:09:25 49352:49352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:25.2432199Z STAGE:2022-09-27 16:09:25 49351:49351 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:25.2443118Z STAGE:2022-09-27 16:09:25 49352:49352 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:09:25 49351:49351 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:25.2443533Z 
2022-09-27T16:09:25.3868162Z ok (4.578s)
2022-09-27T16:09:25.3868392Z 
2022-09-27T16:09:25.3868799Z ----------------------------------------------------------------------
2022-09-27T16:09:25.3869151Z Ran 1 test in 4.578s
2022-09-27T16:09:25.3869314Z 
2022-09-27T16:09:25.3869391Z OK
2022-09-27T16:09:25.3869526Z 
2022-09-27T16:09:25.3869668Z Generating XML reports...
2022-09-27T16:09:25.3905989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160920.xml
2022-09-27T16:09:25.8360010Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:09:25.8360798Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:09:26.3260672Z ok (7.525s)
2022-09-27T16:09:26.3260879Z 
2022-09-27T16:09:26.3261301Z ----------------------------------------------------------------------
2022-09-27T16:09:26.3261642Z Ran 1 test in 7.525s
2022-09-27T16:09:26.3261795Z 
2022-09-27T16:09:26.3261900Z OK
2022-09-27T16:09:26.3262034Z 
2022-09-27T16:09:26.3264626Z Generating XML reports...
2022-09-27T16:09:26.3297861Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160918.xml
2022-09-27T16:09:27.5362410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:27.5364163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:27.5366280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:27.5367419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:28.0466421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:09:28.0489503Z 
2022-09-27T16:09:28.0489814Z Running tests...
2022-09-27T16:09:28.0490267Z ----------------------------------------------------------------------
2022-09-27T16:09:28.4583927Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:28.4584516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:28.4586305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:28.4586932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:28.9658519Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:09:28.9681906Z 
2022-09-27T16:09:28.9682245Z Running tests...
2022-09-27T16:09:28.9682943Z ----------------------------------------------------------------------
2022-09-27T16:09:29.6275353Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:29.6475795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49674
2022-09-27T16:09:29.6482403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49675
2022-09-27T16:09:30.6109820Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:30.6301324Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49739
2022-09-27T16:09:30.6307687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49740
2022-09-27T16:09:31.2596822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:31.2598016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:31.2599853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:31.2601460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:31.4075962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:31.4077542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:31.4079755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:31.4081417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:31.9479379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:31.9480801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:31.9482184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:31.9483507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:31.9487177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:31.9488544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:31.9595991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:09:31.9596562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:09:31.9597303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:09:31.9598013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:09:32.2629825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:32.2630360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:32.2630975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:32.2631463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:32.3046847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:32.3047345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:32.3049650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:32.3050125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:32.4553143Z ok (4.406s)
2022-09-27T16:09:32.4553348Z 
2022-09-27T16:09:32.4553750Z ----------------------------------------------------------------------
2022-09-27T16:09:32.4554088Z Ran 1 test in 4.406s
2022-09-27T16:09:32.4554237Z 
2022-09-27T16:09:32.4554333Z OK
2022-09-27T16:09:32.4554469Z 
2022-09-27T16:09:32.4554849Z Generating XML reports...
2022-09-27T16:09:32.4591540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160928.xml
2022-09-27T16:09:32.7749300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:32.8186378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:32.8187213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:32.8255457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:32.8262220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:32.8327939Z STAGE:2022-09-27 16:09:32 49739:49739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:32.9197619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:32.9261287Z STAGE:2022-09-27 16:09:32 49740:49740 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:34.5020831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:34.5021378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:34.5022900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:34.5023391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:34.5471426Z [1664294974.546659] [28fd6574525b:49739:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:09:34.5471936Z [1664294974.546659] [28fd6574525b:49739:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:09:34.5855513Z [1664294974.585107] [28fd6574525b:49740:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:09:34.5856021Z [1664294974.585107] [28fd6574525b:49740:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:09:34.9676949Z STAGE:2022-09-27 16:09:34 49740:49740 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:09:34 49739:49739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:34.9677389Z 
2022-09-27T16:09:35.0238775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:09:35.0262543Z 
2022-09-27T16:09:35.0262937Z Running tests...
2022-09-27T16:09:35.0263736Z ----------------------------------------------------------------------
2022-09-27T16:09:35.1137857Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:09:35 49740:49740 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:35.1193176Z STAGE:2022-09-27 16:09:35 49739:49739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:35.1203624Z STAGE:2022-09-27 16:09:35 49740:49740 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:35.1204225Z STAGE:2022-09-27 16:09:35 49739:49739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:36.4448422Z ok (7.476s)
2022-09-27T16:09:36.4448657Z 
2022-09-27T16:09:36.4449044Z ----------------------------------------------------------------------
2022-09-27T16:09:36.4449368Z Ran 1 test in 7.477s
2022-09-27T16:09:36.4449534Z 
2022-09-27T16:09:36.4449630Z OK
2022-09-27T16:09:36.4449765Z 
2022-09-27T16:09:36.4449918Z Generating XML reports...
2022-09-27T16:09:36.4505437Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160928.xml
2022-09-27T16:09:36.7454443Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:36.7656844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49871
2022-09-27T16:09:36.7663138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49872
2022-09-27T16:09:38.4401513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:38.4402052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:38.4402929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:38.4403409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:38.4638434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:38.4639129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:38.4642028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:38.4642557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:38.5868737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:38.5869251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:38.5872695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:38.5873227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:39.0018982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:39.0019667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:39.0020693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:39.0021503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:39.0127390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:39.0787582Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:09:39.0810580Z 
2022-09-27T16:09:39.0811083Z Running tests...
2022-09-27T16:09:39.0811568Z ----------------------------------------------------------------------
2022-09-27T16:09:39.0831862Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-09-27T16:09:39.0832620Z 
2022-09-27T16:09:39.0833218Z ----------------------------------------------------------------------
2022-09-27T16:09:39.0833839Z Ran 1 test in 0.002s
2022-09-27T16:09:39.0834030Z 
2022-09-27T16:09:39.0834141Z OK (skipped=1)
2022-09-27T16:09:39.0834296Z 
2022-09-27T16:09:39.0834426Z Generating XML reports...
2022-09-27T16:09:39.0869022Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160939.xml
2022-09-27T16:09:39.1031908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:40.3963778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3qe29iz9
2022-09-27T16:09:40.3965207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3qe29iz9/_remote_module_non_scriptable.py
2022-09-27T16:09:40.9972817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:40.9973349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:40.9975880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:40.9976620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:41.5139942Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:09:41.5163138Z 
2022-09-27T16:09:41.5163358Z Running tests...
2022-09-27T16:09:41.5163801Z ----------------------------------------------------------------------
2022-09-27T16:09:42.2364652Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... ok (23.074s)
2022-09-27T16:09:42.2364934Z 
2022-09-27T16:09:42.2365343Z ----------------------------------------------------------------------
2022-09-27T16:09:42.2365684Z Ran 1 test in 23.074s
2022-09-27T16:09:42.2365851Z 
2022-09-27T16:09:42.2365947Z OK
2022-09-27T16:09:42.2366086Z 
2022-09-27T16:09:42.2366216Z Generating XML reports...
2022-09-27T16:09:42.2401436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160919.xml
2022-09-27T16:09:42.9857340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq984a_7g
2022-09-27T16:09:42.9858227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq984a_7g/_remote_module_non_scriptable.py
2022-09-27T16:09:43.2754621Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:43.2967033Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50016
2022-09-27T16:09:43.2971874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50017
2022-09-27T16:09:43.4579867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:43.4580915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:43.4709830Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:09:43.4713280Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:09:44.1813601Z ok (9.155s)
2022-09-27T16:09:44.1813829Z 
2022-09-27T16:09:44.1814230Z ----------------------------------------------------------------------
2022-09-27T16:09:44.1814580Z Ran 1 test in 9.155s
2022-09-27T16:09:44.1814745Z 
2022-09-27T16:09:44.1814845Z OK
2022-09-27T16:09:44.1814981Z 
2022-09-27T16:09:44.1815117Z Generating XML reports...
2022-09-27T16:09:44.1850583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160935.xml
2022-09-27T16:09:44.5830125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:44.5830657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:44.5831760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:44.5832259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:45.0122371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:45.0122900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:45.0124229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:45.0124722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:45.0468869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:45.0469349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:45.0472930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:45.0473421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:45.0982550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:09:45.1005860Z 
2022-09-27T16:09:45.1006321Z Running tests...
2022-09-27T16:09:45.1006824Z ----------------------------------------------------------------------
2022-09-27T16:09:45.1028244Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:09:45.1028594Z 
2022-09-27T16:09:45.1028867Z ----------------------------------------------------------------------
2022-09-27T16:09:45.1029203Z Ran 1 test in 0.002s
2022-09-27T16:09:45.1029371Z 
2022-09-27T16:09:45.1029466Z OK (skipped=1)
2022-09-27T16:09:45.1029625Z 
2022-09-27T16:09:45.1029750Z Generating XML reports...
2022-09-27T16:09:45.1065480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160945.xml
2022-09-27T16:09:45.5313059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:45.6213502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:45.6214349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:45.6224210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:45.6232002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:45.6299529Z STAGE:2022-09-27 16:09:45 50016:50016 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:45.7223685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:45.7286725Z STAGE:2022-09-27 16:09:45 50017:50017 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:46.2412950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:46.2413968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:46.2415137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:46.2416001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:46.7494082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:09:46.7515990Z 
2022-09-27T16:09:46.7516577Z Running tests...
2022-09-27T16:09:46.7517440Z ----------------------------------------------------------------------
2022-09-27T16:09:47.0921732Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:47.0922954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:47.0924214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:47.0925184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:47.4402274Z [1664294987.439496] [28fd6574525b:50016:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:09:47.4403450Z [1664294987.439496] [28fd6574525b:50016:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:09:47.4470849Z [1664294987.446697] [28fd6574525b:50017:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:09:47.4471917Z [1664294987.446697] [28fd6574525b:50017:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:09:47.6137804Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:09:47.6161763Z 
2022-09-27T16:09:47.6162321Z Running tests...
2022-09-27T16:09:47.6163157Z ----------------------------------------------------------------------
2022-09-27T16:09:47.6188860Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-09-27T16:09:47.6189513Z 
2022-09-27T16:09:47.6190045Z ----------------------------------------------------------------------
2022-09-27T16:09:47.6190664Z Ran 1 test in 0.003s
2022-09-27T16:09:47.6190975Z 
2022-09-27T16:09:47.6191189Z OK (skipped=1)
2022-09-27T16:09:47.6191453Z 
2022-09-27T16:09:47.6191712Z Generating XML reports...
2022-09-27T16:09:47.6227784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160947.xml
2022-09-27T16:09:47.8297719Z STAGE:2022-09-27 16:09:47 50016:50016 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:09:47 50017:50017 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:47.8298579Z 
2022-09-27T16:09:47.9545294Z STAGE:2022-09-27 16:09:47 50017:50017 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:47.9592823Z STAGE:2022-09-27 16:09:47 50016:50016 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:47.9603182Z STAGE:2022-09-27 16:09:47 50017:50017 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:47.9604348Z STAGE:2022-09-27 16:09:47 50016:50016 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:09:48.3105443Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:48.3297858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50202
2022-09-27T16:09:48.3304101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50203
2022-09-27T16:09:49.2110692Z ok (7.694s)
2022-09-27T16:09:49.2111063Z 
2022-09-27T16:09:49.2111756Z ----------------------------------------------------------------------
2022-09-27T16:09:49.2112758Z Ran 1 test in 7.695s
2022-09-27T16:09:49.2113123Z 
2022-09-27T16:09:49.2113297Z OK
2022-09-27T16:09:49.2113546Z 
2022-09-27T16:09:49.2113770Z Generating XML reports...
2022-09-27T16:09:49.2146535Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160941.xml
2022-09-27T16:09:49.7114809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:49.7115760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:49.7116950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:49.7117869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:50.0401387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:50.0402405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:50.0403599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:50.0405001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:50.0719860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:50.0721159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:50.0722845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:50.0723811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:50.2129048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:09:50.2154163Z 
2022-09-27T16:09:50.2154653Z Running tests...
2022-09-27T16:09:50.2155150Z ----------------------------------------------------------------------
2022-09-27T16:09:50.2176053Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:09:50.2176533Z 
2022-09-27T16:09:50.2177035Z ----------------------------------------------------------------------
2022-09-27T16:09:50.2177482Z Ran 1 test in 0.002s
2022-09-27T16:09:50.2177649Z 
2022-09-27T16:09:50.2177744Z OK (skipped=1)
2022-09-27T16:09:50.2177900Z 
2022-09-27T16:09:50.2190167Z Generating XML reports...
2022-09-27T16:09:50.2213876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160950.xml
2022-09-27T16:09:50.6568515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:50.6569119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:50.6569925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:50.6570630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:50.6675895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:50.7581754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:51.3694922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:51.3695681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:51.3697192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:51.3697685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:51.8814781Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:09:51.8837737Z 
2022-09-27T16:09:51.8838216Z Running tests...
2022-09-27T16:09:51.8838778Z ----------------------------------------------------------------------
2022-09-27T16:09:51.8870341Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-09-27T16:09:51.8870694Z 
2022-09-27T16:09:51.8871218Z ----------------------------------------------------------------------
2022-09-27T16:09:51.8871575Z Ran 1 test in 0.003s
2022-09-27T16:09:51.8871745Z 
2022-09-27T16:09:51.8871855Z OK (skipped=1)
2022-09-27T16:09:51.8872025Z 
2022-09-27T16:09:51.8872153Z Generating XML reports...
2022-09-27T16:09:51.8908855Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160951.xml
2022-09-27T16:09:52.0462297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49uphkxv
2022-09-27T16:09:52.0463126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49uphkxv/_remote_module_non_scriptable.py
2022-09-27T16:09:52.1255552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u83p8ve
2022-09-27T16:09:52.1256874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u83p8ve/_remote_module_non_scriptable.py
2022-09-27T16:09:52.2741859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:52.2742543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:52.2744772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:52.2745271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:52.5549088Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:09:52.5754985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:52.5755505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:09:52.7927585Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:09:52.7951856Z 
2022-09-27T16:09:52.7952139Z Running tests...
2022-09-27T16:09:52.7952626Z ----------------------------------------------------------------------
2022-09-27T16:09:52.7974345Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:09:52.7974696Z 
2022-09-27T16:09:52.7974965Z ----------------------------------------------------------------------
2022-09-27T16:09:52.7975286Z Ran 1 test in 0.002s
2022-09-27T16:09:52.7975451Z 
2022-09-27T16:09:52.7975561Z OK (skipped=1)
2022-09-27T16:09:52.7975717Z 
2022-09-27T16:09:52.7975844Z Generating XML reports...
2022-09-27T16:09:52.8011640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160952.xml
2022-09-27T16:09:53.1410091Z ok (6.389s)
2022-09-27T16:09:53.1410297Z 
2022-09-27T16:09:53.1410917Z ----------------------------------------------------------------------
2022-09-27T16:09:53.1411285Z Ran 1 test in 6.389s
2022-09-27T16:09:53.1411455Z 
2022-09-27T16:09:53.1411547Z OK
2022-09-27T16:09:53.1411682Z 
2022-09-27T16:09:53.1411815Z Generating XML reports...
2022-09-27T16:09:53.1448190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160946.xml
2022-09-27T16:09:53.9122008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:53.9122826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:53.9125142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:53.9125636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:54.3948992Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:09:54.3972732Z 
2022-09-27T16:09:54.3973323Z Running tests...
2022-09-27T16:09:54.3974434Z ----------------------------------------------------------------------
2022-09-27T16:09:54.7921539Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:54.7922671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:54.7923876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:54.7925215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:55.2608527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:55.2609494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:55.2610679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:55.2611644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:55.2924471Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:09:55.2947181Z 
2022-09-27T16:09:55.2947598Z Running tests...
2022-09-27T16:09:55.2968766Z ----------------------------------------------------------------------
2022-09-27T16:09:55.2970072Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:09:55.2970758Z 
2022-09-27T16:09:55.2971285Z ----------------------------------------------------------------------
2022-09-27T16:09:55.2971918Z Ran 1 test in 0.002s
2022-09-27T16:09:55.2972243Z 
2022-09-27T16:09:55.2972451Z OK (skipped=1)
2022-09-27T16:09:55.2972764Z 
2022-09-27T16:09:55.2973003Z Generating XML reports...
2022-09-27T16:09:55.3003413Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160955.xml
2022-09-27T16:09:55.7793165Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:09:55.7815922Z 
2022-09-27T16:09:55.7816261Z Running tests...
2022-09-27T16:09:55.7817145Z ----------------------------------------------------------------------
2022-09-27T16:09:55.9552519Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:55.9749157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50483
2022-09-27T16:09:55.9755021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50484
2022-09-27T16:09:57.3618330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:57.3618830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:57.3619830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:57.3620322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:57.4967441Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:57.5164675Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50580
2022-09-27T16:09:57.5171254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50581
2022-09-27T16:09:57.6764544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:57.6765495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:57.6766641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:57.6767595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:57.7429571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:57.7430084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:57.7432333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:57.7432836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:57.8562579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:09:57.8585116Z 
2022-09-27T16:09:57.8585327Z Running tests...
2022-09-27T16:09:57.8585774Z ----------------------------------------------------------------------
2022-09-27T16:09:58.1925482Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:58.3151799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:58.3152677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:58.3244379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:58.3251679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:58.3321673Z STAGE:2022-09-27 16:09:58 50483:50483 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:58.4164698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:09:58.4230851Z STAGE:2022-09-27 16:09:58 50484:50484 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:09:59.1632592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:59.1633576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:59.1634783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:59.1635342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:59.1977598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:09:59.1978134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:09:59.1980307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:09:59.1980814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:09:59.5367056Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:09:59.5569722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50660
2022-09-27T16:09:59.5574945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50661
2022-09-27T16:09:59.7460899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:09:59.7461687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:09:59.7462531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:59.7463250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:09:59.7469882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:09:59.7470360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:00.1055026Z [1664295000.104931] [28fd6574525b:50483:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:10:00.1055576Z [1664295000.104931] [28fd6574525b:50483:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:10:00.1371059Z [1664295000.136645] [28fd6574525b:50484:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:10:00.1371595Z [1664295000.136645] [28fd6574525b:50484:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:10:00.2258180Z ok (4.443s)
2022-09-27T16:10:00.2258635Z 
2022-09-27T16:10:00.2261026Z ----------------------------------------------------------------------
2022-09-27T16:10:00.2262396Z Ran 1 test in 4.444s
2022-09-27T16:10:00.2263173Z 
2022-09-27T16:10:00.2263636Z OK
2022-09-27T16:10:00.2264243Z 
2022-09-27T16:10:00.2264795Z Generating XML reports...
2022-09-27T16:10:00.2317442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160955.xml
2022-09-27T16:10:00.6014395Z STAGE:2022-09-27 16:10:00 50484:50484 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:00.6032469Z STAGE:2022-09-27 16:10:00 50483:50483 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:00.7299375Z STAGE:2022-09-27 16:10:00 50484:50484 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:00.7312397Z STAGE:2022-09-27 16:10:00 50483:50483 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:00.7323736Z STAGE:2022-09-27 16:10:00 50484:50484 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:00.7324325Z STAGE:2022-09-27 16:10:00 50483:50483 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:01.2268300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:01.2268812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:01.2271508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:01.2271985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:01.2964315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:01.2964842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:01.2966357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:01.2966853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:01.7903961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:01.8757187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:01.8757998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:01.8814835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:01.8821889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:01.8883067Z ok (7.491s)
2022-09-27T16:10:01.8883513Z 
2022-09-27T16:10:01.8883959Z ----------------------------------------------------------------------
2022-09-27T16:10:01.8884541Z Ran 1 test in 7.491s
2022-09-27T16:10:01.8884715Z 
2022-09-27T16:10:01.8884820Z OK
2022-09-27T16:10:01.8884958Z 
2022-09-27T16:10:01.8885074Z Generating XML reports...
2022-09-27T16:10:01.8920977Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160954.xml
2022-09-27T16:10:01.9771912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:02.5432479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:02.5433437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:02.5434623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:02.5435565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:03.1192366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:03.1238468Z 
2022-09-27T16:10:03.1241915Z Running tests...
2022-09-27T16:10:03.1246004Z ----------------------------------------------------------------------
2022-09-27T16:10:03.9674574Z   test_gather (__main__.TestDistBackendWithSpawn) ... ok (6.109s)
2022-09-27T16:10:03.9674974Z 
2022-09-27T16:10:03.9675633Z ----------------------------------------------------------------------
2022-09-27T16:10:03.9676049Z Ran 1 test in 6.109s
2022-09-27T16:10:03.9676200Z 
2022-09-27T16:10:03.9676295Z OK
2022-09-27T16:10:03.9676430Z 
2022-09-27T16:10:03.9676558Z Generating XML reports...
2022-09-27T16:10:03.9711252Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160957.xml
2022-09-27T16:10:04.0782421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:04.0782984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:04.0784743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:04.0785241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:04.5788093Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:04.5810533Z 
2022-09-27T16:10:04.5810680Z Running tests...
2022-09-27T16:10:04.5811593Z ----------------------------------------------------------------------
2022-09-27T16:10:04.5832346Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:10:04.5832674Z 
2022-09-27T16:10:04.5832993Z ----------------------------------------------------------------------
2022-09-27T16:10:04.5833310Z Ran 1 test in 0.002s
2022-09-27T16:10:04.5833481Z 
2022-09-27T16:10:04.5833591Z OK (skipped=1)
2022-09-27T16:10:04.5833759Z 
2022-09-27T16:10:04.5833884Z Generating XML reports...
2022-09-27T16:10:04.5868595Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161004.xml
2022-09-27T16:10:04.7997826Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:04.8199289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50823
2022-09-27T16:10:04.8205722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50824
2022-09-27T16:10:06.1113355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:06.1113858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:06.1114788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:06.1115309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:06.4682649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:06.4683164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:06.4684580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:06.4685241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:06.5253898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:06.5254392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:06.5258396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:06.5259010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:06.5832745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:06.5833538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:06.5835032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:06.5835531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:06.5995544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:06.6019077Z 
2022-09-27T16:10:06.6019218Z Running tests...
2022-09-27T16:10:06.6019925Z ----------------------------------------------------------------------
2022-09-27T16:10:07.0565638Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:07.0566304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:07.0567122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:07.0567837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:07.0674854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:07.0743525Z STAGE:2022-09-27 16:10:07 50824:50824 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:07.0872131Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:07.0896687Z 
2022-09-27T16:10:07.0896898Z Running tests...
2022-09-27T16:10:07.0897356Z ----------------------------------------------------------------------
2022-09-27T16:10:07.0928703Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.003s)
2022-09-27T16:10:07.0929042Z 
2022-09-27T16:10:07.0929329Z ----------------------------------------------------------------------
2022-09-27T16:10:07.0929661Z Ran 1 test in 0.003s
2022-09-27T16:10:07.0929828Z 
2022-09-27T16:10:07.0929939Z OK (skipped=1)
2022-09-27T16:10:07.0930100Z 
2022-09-27T16:10:07.0930210Z Generating XML reports...
2022-09-27T16:10:07.0965360Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161007.xml
2022-09-27T16:10:07.1580135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:07.1646000Z STAGE:2022-09-27 16:10:07 50823:50823 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:07.1661181Z STAGE:2022-09-27 16:10:07 50824:50824 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:07.1661829Z STAGE:2022-09-27 16:10:07 50823:50823 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:07.1674346Z STAGE:2022-09-27 16:10:07 50824:50824 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:07.1684149Z STAGE:2022-09-27 16:10:07 50823:50823 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:07.1696011Z STAGE:2022-09-27 16:10:07 50823:50823 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:07.1696620Z STAGE:2022-09-27 16:10:07 50824:50824 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:07.7278085Z ok (4.604s)
2022-09-27T16:10:07.7278505Z 
2022-09-27T16:10:07.7279272Z ----------------------------------------------------------------------
2022-09-27T16:10:07.7279900Z Ran 1 test in 4.604s
2022-09-27T16:10:07.7280076Z 
2022-09-27T16:10:07.7280446Z OK
2022-09-27T16:10:07.7280582Z 
2022-09-27T16:10:07.7280737Z Generating XML reports...
2022-09-27T16:10:07.7315179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161003.xml
2022-09-27T16:10:08.2315715Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:08.2514580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51001
2022-09-27T16:10:08.2520127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51002
2022-09-27T16:10:09.1625469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:09.1625975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:09.1626969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:09.1627461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:09.6661127Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:09.6685389Z 
2022-09-27T16:10:09.6685729Z Running tests...
2022-09-27T16:10:09.6686167Z ----------------------------------------------------------------------
2022-09-27T16:10:09.8532471Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:09.8533050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:09.8533961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:09.8534432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:09.8555936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:09.8556420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:09.8560519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:09.8561249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:09.9135525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:09.9136013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:09.9138315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:09.9138784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:10.3607495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:10.3629684Z 
2022-09-27T16:10:10.3629829Z Running tests...
2022-09-27T16:10:10.3630559Z ----------------------------------------------------------------------
2022-09-27T16:10:10.3762897Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:10.4263539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:10.4264351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:10.4272015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:10.4278665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:10.5277929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:11.3265740Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:11.3478500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51111
2022-09-27T16:10:11.3484309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51112
2022-09-27T16:10:12.1067398Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:12.1493222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51183
2022-09-27T16:10:12.1544106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51184
2022-09-27T16:10:12.4614421Z ok (5.859s)
2022-09-27T16:10:12.4614855Z 
2022-09-27T16:10:12.4615617Z ----------------------------------------------------------------------
2022-09-27T16:10:12.4616049Z Ran 1 test in 5.859s
2022-09-27T16:10:12.4616219Z 
2022-09-27T16:10:12.4616314Z OK
2022-09-27T16:10:12.4616453Z 
2022-09-27T16:10:12.4616590Z Generating XML reports...
2022-09-27T16:10:12.4652054Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161006.xml
2022-09-27T16:10:13.0987660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:13.0988641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:13.0989868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:13.0990391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:13.1527528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:13.1528083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:13.1530015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:13.1530501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:13.6267493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:13.7271028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:13.7271896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:13.7284534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:13.7293977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:13.7396821Z STAGE:2022-09-27 16:10:13 51111:51111 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:13.7848397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:13.7848860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:13.7853374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:13.7853895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:13.8213240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:13.8213704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:13.8217514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:13.8217996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:13.8281221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:13.8343552Z STAGE:2022-09-27 16:10:13 51112:51112 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:14.3582670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:14.3583256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:14.3584030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:14.3584958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:14.3592028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:14.3592562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:14.6189986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:14.6190508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:14.6191830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:14.6192328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:14.7613865Z ok (4.398s)
2022-09-27T16:10:14.7614108Z 
2022-09-27T16:10:14.7614514Z ----------------------------------------------------------------------
2022-09-27T16:10:14.7614878Z Ran 1 test in 4.398s
2022-09-27T16:10:14.7615051Z 
2022-09-27T16:10:14.7615129Z OK
2022-09-27T16:10:14.7615269Z 
2022-09-27T16:10:14.7615407Z Generating XML reports...
2022-09-27T16:10:14.7651913Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161010.xml
2022-09-27T16:10:15.1599675Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:15.1623773Z 
2022-09-27T16:10:15.1624288Z Running tests...
2022-09-27T16:10:15.1624807Z ----------------------------------------------------------------------
2022-09-27T16:10:15.5343934Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... [1664295015.533902] [28fd6574525b:51112:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:10:15.5344560Z [1664295015.533902] [28fd6574525b:51112:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:10:15.5474059Z [1664295015.547001] [28fd6574525b:51111:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:10:15.5474567Z [1664295015.547001] [28fd6574525b:51111:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:10:15.9553356Z STAGE:2022-09-27 16:10:15 51112:51112 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:15.9555481Z STAGE:2022-09-27 16:10:15 51111:51111 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:16.1263343Z STAGE:2022-09-27 16:10:16 51112:51112 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:16.1288152Z STAGE:2022-09-27 16:10:16 51111:51111 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:16.1299380Z STAGE:2022-09-27 16:10:16 51112:51112 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:10:16 51111:51111 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:16.1299808Z 
2022-09-27T16:10:16.8410722Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:16.8607075Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51341
2022-09-27T16:10:16.8614235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51342
2022-09-27T16:10:16.9418861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:16.9419393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:16.9421600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:16.9422097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:17.2616180Z ok (7.593s)
2022-09-27T16:10:17.2616678Z 
2022-09-27T16:10:17.2617088Z ----------------------------------------------------------------------
2022-09-27T16:10:17.2617436Z Ran 1 test in 7.593s
2022-09-27T16:10:17.2617605Z 
2022-09-27T16:10:17.2617682Z OK
2022-09-27T16:10:17.2617823Z 
2022-09-27T16:10:17.2617957Z Generating XML reports...
2022-09-27T16:10:17.2653717Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161009.xml
2022-09-27T16:10:17.4436333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:17.4459517Z 
2022-09-27T16:10:17.4459812Z Running tests...
2022-09-27T16:10:17.4460235Z ----------------------------------------------------------------------
2022-09-27T16:10:17.4479052Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:10:17.4479375Z 
2022-09-27T16:10:17.4479659Z ----------------------------------------------------------------------
2022-09-27T16:10:17.4479987Z Ran 1 test in 0.002s
2022-09-27T16:10:17.4480430Z 
2022-09-27T16:10:17.4480550Z OK (skipped=1)
2022-09-27T16:10:17.4480711Z 
2022-09-27T16:10:17.4480840Z Generating XML reports...
2022-09-27T16:10:17.4517241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161017.xml
2022-09-27T16:10:18.5686424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:18.5686917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:18.5689127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:18.5689650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:18.5884003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:18.5884477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:18.5887776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:18.5888271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:19.0761984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:19.1072408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:19.1073241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:19.1167473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:19.1173895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:19.2085835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:19.3738867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:19.3739373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:19.3742049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:19.3742567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:19.3759560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:19.3760051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:19.3764822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:19.3765567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:19.8957959Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:19.8958580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:19.8981376Z 
2022-09-27T16:10:19.8981916Z Running tests...
2022-09-27T16:10:19.8982433Z ----------------------------------------------------------------------
2022-09-27T16:10:19.8982688Z 
2022-09-27T16:10:19.8982816Z Running tests...
2022-09-27T16:10:19.8983211Z ----------------------------------------------------------------------
2022-09-27T16:10:19.8999847Z   test_scatter (__main__.TestDistBackendWithSpawn) ...   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:10:19.9001047Z 
2022-09-27T16:10:19.9001551Z ----------------------------------------------------------------------
2022-09-27T16:10:19.9001904Z Ran 1 test in 0.002s
2022-09-27T16:10:19.9002077Z 
2022-09-27T16:10:19.9002190Z OK (skipped=1)
2022-09-27T16:10:19.9002330Z 
2022-09-27T16:10:19.9002459Z Generating XML reports...
2022-09-27T16:10:19.9035749Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161019.xml
2022-09-27T16:10:21.5798324Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:21.6003947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51522
2022-09-27T16:10:21.6009790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51523
2022-09-27T16:10:21.9259942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:21.9260462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:21.9261945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:21.9262447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:22.1728507Z ok (7.010s)
2022-09-27T16:10:22.1728727Z 
2022-09-27T16:10:22.1729093Z ----------------------------------------------------------------------
2022-09-27T16:10:22.1729439Z Ran 1 test in 7.010s
2022-09-27T16:10:22.1729614Z 
2022-09-27T16:10:22.1729711Z OK
2022-09-27T16:10:22.1729849Z 
2022-09-27T16:10:22.1729988Z Generating XML reports...
2022-09-27T16:10:22.1765752Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161015.xml
2022-09-27T16:10:22.4381887Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:22.4412675Z 
2022-09-27T16:10:22.4413255Z Running tests...
2022-09-27T16:10:22.4414141Z ----------------------------------------------------------------------
2022-09-27T16:10:22.4437804Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:10:22.4438294Z 
2022-09-27T16:10:22.4438607Z ----------------------------------------------------------------------
2022-09-27T16:10:22.4438962Z Ran 1 test in 0.003s
2022-09-27T16:10:22.4439110Z 
2022-09-27T16:10:22.4439226Z OK (skipped=1)
2022-09-27T16:10:22.4439383Z 
2022-09-27T16:10:22.4439512Z Generating XML reports...
2022-09-27T16:10:22.4475849Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161022.xml
2022-09-27T16:10:23.2225153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:23.2225697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:23.2228217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:23.2228723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:23.2250244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:23.2250729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:23.2254239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:23.2254744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:23.8198357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:23.8198898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:23.8199733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:23.8200705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:23.8306021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:23.9209565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:23.9319579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:10:23.9320131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:10:23.9321167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:10:23.9321883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:10:23.9414860Z STAGE:2022-09-27 16:10:23 51523:51523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:23.9415461Z STAGE:2022-09-27 16:10:23 51522:51522 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:23.9429419Z STAGE:2022-09-27 16:10:23 51523:51523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:23.9430020Z STAGE:2022-09-27 16:10:23 51522:51522 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:23.9441089Z STAGE:2022-09-27 16:10:23 51523:51523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:23.9453421Z STAGE:2022-09-27 16:10:23 51522:51522 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:10:23.9465520Z STAGE:2022-09-27 16:10:23 51522:51522 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:23.9466337Z STAGE:2022-09-27 16:10:23 51523:51523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:10:24.3074704Z ok (4.409s)
2022-09-27T16:10:24.3075047Z 
2022-09-27T16:10:24.3075458Z ----------------------------------------------------------------------
2022-09-27T16:10:24.3075800Z Ran 1 test in 4.409s
2022-09-27T16:10:24.3075968Z 
2022-09-27T16:10:24.3076063Z OK
2022-09-27T16:10:24.3076198Z 
2022-09-27T16:10:24.3076332Z Generating XML reports...
2022-09-27T16:10:24.3111857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161019.xml
2022-09-27T16:10:24.3421633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:24.3422159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:24.3424031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:24.3424745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:24.4275067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:24.4275605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:24.4277148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:24.4277773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:24.8410243Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:24.8432403Z 
2022-09-27T16:10:24.8432522Z Running tests...
2022-09-27T16:10:24.8433674Z ----------------------------------------------------------------------
2022-09-27T16:10:24.9515603Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:24.9538523Z 
2022-09-27T16:10:24.9538768Z Running tests...
2022-09-27T16:10:24.9539212Z ----------------------------------------------------------------------
2022-09-27T16:10:24.9558267Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:10:24.9558619Z 
2022-09-27T16:10:24.9558906Z ----------------------------------------------------------------------
2022-09-27T16:10:24.9559225Z Ran 1 test in 0.002s
2022-09-27T16:10:24.9559392Z 
2022-09-27T16:10:24.9559507Z OK (skipped=1)
2022-09-27T16:10:24.9559664Z 
2022-09-27T16:10:24.9559790Z Generating XML reports...
2022-09-27T16:10:24.9594200Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161024.xml
2022-09-27T16:10:26.5132026Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:26.5284941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:26.5285452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:26.5289340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:26.5289828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:26.5330316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51741
2022-09-27T16:10:26.5338133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51742
2022-09-27T16:10:26.9803192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:26.9803719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:26.9806300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:26.9806803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:27.0350236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:27.0373784Z 
2022-09-27T16:10:27.0374116Z Running tests...
2022-09-27T16:10:27.0374573Z ----------------------------------------------------------------------
2022-09-27T16:10:27.5322943Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:27.5345477Z 
2022-09-27T16:10:27.5345709Z Running tests...
2022-09-27T16:10:27.5346314Z ----------------------------------------------------------------------
2022-09-27T16:10:27.5364637Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:10:27.5365206Z 
2022-09-27T16:10:27.5365684Z ----------------------------------------------------------------------
2022-09-27T16:10:27.5366079Z Ran 1 test in 0.002s
2022-09-27T16:10:27.5366249Z 
2022-09-27T16:10:27.5366365Z OK (skipped=1)
2022-09-27T16:10:27.5366541Z 
2022-09-27T16:10:27.5366671Z Generating XML reports...
2022-09-27T16:10:27.5400407Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161027.xml
2022-09-27T16:10:28.1899170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:28.1899657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:28.1900775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:28.1901263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:28.1993625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:28.1994078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:28.1997785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:28.1998275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:28.7030039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:28.7092898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:28.7094387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:28.7130522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:28.7137421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:28.7296408Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:28.7501095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51849
2022-09-27T16:10:28.7507286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51850
2022-09-27T16:10:28.8109161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:29.4714134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:29.4715287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:29.4718870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:29.4720542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:29.9861369Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:29.9884167Z 
2022-09-27T16:10:29.9884506Z Running tests...
2022-09-27T16:10:29.9884949Z ----------------------------------------------------------------------
2022-09-27T16:10:29.9903357Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:10:29.9904048Z 
2022-09-27T16:10:29.9904372Z ----------------------------------------------------------------------
2022-09-27T16:10:29.9904726Z Ran 1 test in 0.002s
2022-09-27T16:10:29.9904895Z 
2022-09-27T16:10:29.9905007Z OK (skipped=1)
2022-09-27T16:10:29.9905149Z 
2022-09-27T16:10:29.9905276Z Generating XML reports...
2022-09-27T16:10:29.9938992Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161029.xml
2022-09-27T16:10:30.4036739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:30.4037703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:30.4038871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:30.4039804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:30.4126102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:30.4127499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:30.4129176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:30.4130117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:30.7432294Z ok (5.900s)
2022-09-27T16:10:30.7432661Z 
2022-09-27T16:10:30.7433338Z ----------------------------------------------------------------------
2022-09-27T16:10:30.7434002Z Ran 1 test in 5.900s
2022-09-27T16:10:30.7434321Z 
2022-09-27T16:10:30.7434475Z OK
2022-09-27T16:10:30.7434738Z 
2022-09-27T16:10:30.7434984Z Generating XML reports...
2022-09-27T16:10:30.7469713Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161024.xml
2022-09-27T16:10:30.9681602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:30.9683531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:30.9686905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:30.9688981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:30.9833875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:31.0747927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:31.4570716Z skip: Skipped due to small world size. (4.419s)
2022-09-27T16:10:31.4570982Z 
2022-09-27T16:10:31.4571379Z ----------------------------------------------------------------------
2022-09-27T16:10:31.4571901Z Ran 1 test in 4.420s
2022-09-27T16:10:31.4572197Z 
2022-09-27T16:10:31.4572304Z OK (skipped=1)
2022-09-27T16:10:31.4572481Z 
2022-09-27T16:10:31.4575960Z Generating XML reports...
2022-09-27T16:10:31.4607774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161027.xml
2022-09-27T16:10:32.0683483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:32.0683989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:32.0686557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:32.0687065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:32.6325278Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:32.6367633Z 
2022-09-27T16:10:32.6369075Z Running tests...
2022-09-27T16:10:32.6370838Z ----------------------------------------------------------------------
2022-09-27T16:10:32.6405786Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.003s)
2022-09-27T16:10:32.6406507Z 
2022-09-27T16:10:32.6407136Z ----------------------------------------------------------------------
2022-09-27T16:10:32.6407861Z Ran 1 test in 0.004s
2022-09-27T16:10:32.6408248Z 
2022-09-27T16:10:32.6408502Z OK (skipped=1)
2022-09-27T16:10:32.6408837Z 
2022-09-27T16:10:32.6409115Z Generating XML reports...
2022-09-27T16:10:32.6468648Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161032.xml
2022-09-27T16:10:32.9108597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:32.9109181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:32.9110471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:32.9111263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:33.3911544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:33.3933648Z 
2022-09-27T16:10:33.3933917Z Running tests...
2022-09-27T16:10:33.3934370Z ----------------------------------------------------------------------
2022-09-27T16:10:33.5686064Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:33.5696460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:33.5698012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:33.5699171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:34.0601946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:34.0624528Z 
2022-09-27T16:10:34.0624942Z Running tests...
2022-09-27T16:10:34.0625480Z ----------------------------------------------------------------------
2022-09-27T16:10:34.6134759Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:34.6135368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:34.6136871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:34.6137342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:35.0356709Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:35.0528132Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.659s)
2022-09-27T16:10:35.0528767Z 
2022-09-27T16:10:35.0529054Z ----------------------------------------------------------------------
2022-09-27T16:10:35.0529379Z Ran 1 test in 1.659s
2022-09-27T16:10:35.0529544Z 
2022-09-27T16:10:35.0529653Z OK (skipped=1)
2022-09-27T16:10:35.0529809Z 
2022-09-27T16:10:35.0529935Z Generating XML reports...
2022-09-27T16:10:35.0563084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161033.xml
2022-09-27T16:10:35.1351527Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:35.1375433Z 
2022-09-27T16:10:35.1375985Z Running tests...
2022-09-27T16:10:35.1376446Z ----------------------------------------------------------------------
2022-09-27T16:10:35.1396138Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:10:35.1396502Z 
2022-09-27T16:10:35.1396821Z ----------------------------------------------------------------------
2022-09-27T16:10:35.1397159Z Ran 1 test in 0.002s
2022-09-27T16:10:35.1397323Z 
2022-09-27T16:10:35.1397433Z OK (skipped=1)
2022-09-27T16:10:35.1397590Z 
2022-09-27T16:10:35.1397717Z Generating XML reports...
2022-09-27T16:10:35.1432715Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161035.xml
2022-09-27T16:10:35.6320911Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:35.6516025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52070
2022-09-27T16:10:35.6522054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52071
2022-09-27T16:10:37.2982282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:37.2983141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:37.2984050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:37.2984522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:37.3491273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:37.3491773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:37.3493806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:37.3494288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:37.4448528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:37.4449078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:37.4451346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:37.4451822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:37.5295764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:37.5296253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:37.5298334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:37.5298811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:37.7947820Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:37.7971651Z 
2022-09-27T16:10:37.7971902Z Running tests...
2022-09-27T16:10:37.7972331Z ----------------------------------------------------------------------
2022-09-27T16:10:37.7995563Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:10:37.7995887Z 
2022-09-27T16:10:37.7996158Z ----------------------------------------------------------------------
2022-09-27T16:10:37.7996497Z Ran 1 test in 0.002s
2022-09-27T16:10:37.7996645Z 
2022-09-27T16:10:37.7996765Z OK (skipped=1)
2022-09-27T16:10:37.7996922Z 
2022-09-27T16:10:37.7997048Z Generating XML reports...
2022-09-27T16:10:37.8031204Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161037.xml
2022-09-27T16:10:37.8444698Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:37.8468881Z 
2022-09-27T16:10:37.8469597Z Running tests...
2022-09-27T16:10:37.8470523Z ----------------------------------------------------------------------
2022-09-27T16:10:37.8492976Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:10:37.8493620Z 
2022-09-27T16:10:37.8494197Z ----------------------------------------------------------------------
2022-09-27T16:10:37.8494842Z Ran 1 test in 0.002s
2022-09-27T16:10:37.8495153Z 
2022-09-27T16:10:37.8495343Z OK (skipped=1)
2022-09-27T16:10:37.8495630Z 
2022-09-27T16:10:37.8495850Z Generating XML reports...
2022-09-27T16:10:37.8532578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161037.xml
2022-09-27T16:10:38.0663852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:38.0664944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:38.0666376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:38.0668077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:38.0672260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:38.0673227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:38.4590757Z ok (4.396s)
2022-09-27T16:10:38.4590954Z 
2022-09-27T16:10:38.4591586Z ----------------------------------------------------------------------
2022-09-27T16:10:38.4591946Z Ran 1 test in 4.396s
2022-09-27T16:10:38.4592114Z 
2022-09-27T16:10:38.4592209Z OK
2022-09-27T16:10:38.4592343Z 
2022-09-27T16:10:38.4592461Z Generating XML reports...
2022-09-27T16:10:38.4627315Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161034.xml
2022-09-27T16:10:39.7571327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:39.7571876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:39.7574267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:39.7574750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:39.7738304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:39.7738778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:39.7742801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:39.7743292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:40.2391248Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:40.2414252Z 
2022-09-27T16:10:40.2414754Z Running tests...
2022-09-27T16:10:40.2415249Z ----------------------------------------------------------------------
2022-09-27T16:10:40.2438291Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:10:40.2438787Z 
2022-09-27T16:10:40.2439100Z ----------------------------------------------------------------------
2022-09-27T16:10:40.2439455Z Ran 1 test in 0.002s
2022-09-27T16:10:40.2439623Z 
2022-09-27T16:10:40.2439719Z OK (skipped=1)
2022-09-27T16:10:40.2439881Z 
2022-09-27T16:10:40.2440006Z Generating XML reports...
2022-09-27T16:10:40.2475221Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161040.xml
2022-09-27T16:10:40.2727852Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:40.2751662Z 
2022-09-27T16:10:40.2752247Z Running tests...
2022-09-27T16:10:40.2752770Z ----------------------------------------------------------------------
2022-09-27T16:10:40.5066370Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:40.5067186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:40.5068272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:40.5068955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:41.0096193Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:41.0117976Z 
2022-09-27T16:10:41.0118258Z Running tests...
2022-09-27T16:10:41.0118726Z ----------------------------------------------------------------------
2022-09-27T16:10:41.9999005Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:42.0202835Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52342
2022-09-27T16:10:42.0208728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52343
2022-09-27T16:10:42.2243232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:42.2243778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:42.2245242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:42.2245736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:42.6472740Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:42.6772404Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.664s)
2022-09-27T16:10:42.6774367Z 
2022-09-27T16:10:42.6775435Z ----------------------------------------------------------------------
2022-09-27T16:10:42.6776185Z Ran 1 test in 1.665s
2022-09-27T16:10:42.6776532Z 
2022-09-27T16:10:42.6776768Z OK (skipped=1)
2022-09-27T16:10:42.6777084Z 
2022-09-27T16:10:42.6777344Z Generating XML reports...
2022-09-27T16:10:42.6826254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161041.xml
2022-09-27T16:10:42.7799935Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:42.7822143Z 
2022-09-27T16:10:42.7822398Z Running tests...
2022-09-27T16:10:42.7822866Z ----------------------------------------------------------------------
2022-09-27T16:10:42.7844295Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:10:42.7844680Z 
2022-09-27T16:10:42.7844964Z ----------------------------------------------------------------------
2022-09-27T16:10:42.7845281Z Ran 1 test in 0.002s
2022-09-27T16:10:42.7845448Z 
2022-09-27T16:10:42.7845557Z OK (skipped=1)
2022-09-27T16:10:42.7845718Z 
2022-09-27T16:10:42.7845846Z Generating XML reports...
2022-09-27T16:10:42.7879995Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161042.xml
2022-09-27T16:10:43.7233159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:43.7233673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:43.7235965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:43.7236473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:43.7369626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:43.7370094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:43.7377120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:43.7377618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:44.3268101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:44.3866267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:44.3867878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:44.3875135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:44.3882555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:44.4879211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:44.8354635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:44.8355629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:44.8356847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:44.8357771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:44.8855616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:44.8856579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:44.8857785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:44.8858690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:45.3479406Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:45.3501426Z 
2022-09-27T16:10:45.3501992Z Running tests...
2022-09-27T16:10:45.3502865Z ----------------------------------------------------------------------
2022-09-27T16:10:45.3954449Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:45.3979385Z 
2022-09-27T16:10:45.3979936Z Running tests...
2022-09-27T16:10:45.3980476Z ----------------------------------------------------------------------
2022-09-27T16:10:45.8858798Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... [1664295045.885321] [28fd6574525b:52343:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:10:45.8859420Z [1664295045.885321] [28fd6574525b:52343:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:10:45.8933956Z [1664295045.893066] [28fd6574525b:52342:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:10:45.8934701Z [1664295045.893066] [28fd6574525b:52342:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:10:46.8313979Z ok (6.556s)
2022-09-27T16:10:46.8314215Z 
2022-09-27T16:10:46.8314827Z ----------------------------------------------------------------------
2022-09-27T16:10:46.8315268Z Ran 1 test in 6.556s
2022-09-27T16:10:46.8315441Z 
2022-09-27T16:10:46.8315799Z OK
2022-09-27T16:10:46.8315957Z 
2022-09-27T16:10:46.8350181Z Generating XML reports...
2022-09-27T16:10:46.8351125Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161040.xml
2022-09-27T16:10:47.0081532Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:47.0112244Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:47.0281218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52487
2022-09-27T16:10:47.0287763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52488
2022-09-27T16:10:47.0307656Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52490
2022-09-27T16:10:47.0314241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52491
2022-09-27T16:10:48.7270130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:48.7270664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:48.7271275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:48.7272058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:48.7328957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:48.7329427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:48.7333353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:48.7333828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:48.7588063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:48.7588551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:48.7590854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:48.7591329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:48.8068859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:48.8069345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:48.8071327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:48.8071799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:48.9616981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:48.9617478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:48.9620212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:48.9620700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:49.2452821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:49.2668219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:49.2669038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:49.2757061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:49.2764007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:49.3232103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:49.3232844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:49.3233592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:49.3234297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:49.3340686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:49.3680230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:49.4248907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:49.4458125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:10:49.4458699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:10:49.4459488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:10:49.4460959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:10:49.4514455Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:49.4539431Z 
2022-09-27T16:10:49.4539579Z Running tests...
2022-09-27T16:10:49.4540742Z ----------------------------------------------------------------------
2022-09-27T16:10:49.4558403Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-09-27T16:10:49.4558743Z 
2022-09-27T16:10:49.4559042Z ----------------------------------------------------------------------
2022-09-27T16:10:49.4559378Z Ran 1 test in 0.002s
2022-09-27T16:10:49.4559529Z 
2022-09-27T16:10:49.4559667Z OK (skipped=1)
2022-09-27T16:10:49.4559830Z 
2022-09-27T16:10:49.4559957Z Generating XML reports...
2022-09-27T16:10:49.4595856Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161049.xml
2022-09-27T16:10:49.7378729Z ok (4.387s)
2022-09-27T16:10:49.7379166Z 
2022-09-27T16:10:49.7379843Z ----------------------------------------------------------------------
2022-09-27T16:10:49.7380530Z Ran 1 test in 4.388s
2022-09-27T16:10:49.7380896Z 
2022-09-27T16:10:49.7381075Z OK
2022-09-27T16:10:49.7381347Z 
2022-09-27T16:10:49.7381607Z Generating XML reports...
2022-09-27T16:10:49.7415943Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161045.xml
2022-09-27T16:10:49.9358377Z ok (4.538s)
2022-09-27T16:10:49.9358636Z 
2022-09-27T16:10:49.9359025Z ----------------------------------------------------------------------
2022-09-27T16:10:49.9359367Z Ran 1 test in 4.538s
2022-09-27T16:10:49.9359554Z 
2022-09-27T16:10:49.9359651Z OK
2022-09-27T16:10:49.9359774Z 
2022-09-27T16:10:49.9359911Z Generating XML reports...
2022-09-27T16:10:49.9464762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161045.xml
2022-09-27T16:10:51.6386599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:51.6387116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:51.6387912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:51.6388399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:51.9646763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:51.9647299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:51.9649679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:51.9650201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:52.1381260Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:52.1404479Z 
2022-09-27T16:10:52.1404622Z Running tests...
2022-09-27T16:10:52.1405433Z ----------------------------------------------------------------------
2022-09-27T16:10:52.1423774Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-09-27T16:10:52.1424146Z 
2022-09-27T16:10:52.1424444Z ----------------------------------------------------------------------
2022-09-27T16:10:52.1424769Z Ran 1 test in 0.002s
2022-09-27T16:10:52.1424936Z 
2022-09-27T16:10:52.1425047Z OK (skipped=1)
2022-09-27T16:10:52.1425207Z 
2022-09-27T16:10:52.1425356Z Generating XML reports...
2022-09-27T16:10:52.1460660Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161052.xml
2022-09-27T16:10:52.1527647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:52.1528125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:52.1531339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:52.1531815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:52.4567924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:52.4589537Z 
2022-09-27T16:10:52.4589805Z Running tests...
2022-09-27T16:10:52.4590255Z ----------------------------------------------------------------------
2022-09-27T16:10:52.6518960Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:52.6540799Z 
2022-09-27T16:10:52.6541108Z Running tests...
2022-09-27T16:10:52.6541525Z ----------------------------------------------------------------------
2022-09-27T16:10:54.0752179Z   test_get_future (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:54.0755054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:54.0755662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:54.0756160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:54.1193153Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:54.1392522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52808
2022-09-27T16:10:54.1398547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52809
2022-09-27T16:10:54.2556099Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:54.2752185Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52811
2022-09-27T16:10:54.2757513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52812
2022-09-27T16:10:54.5843618Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:54.5867562Z 
2022-09-27T16:10:54.5868084Z Running tests...
2022-09-27T16:10:54.5868569Z ----------------------------------------------------------------------
2022-09-27T16:10:54.5886763Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-09-27T16:10:54.5887242Z 
2022-09-27T16:10:54.5887667Z ----------------------------------------------------------------------
2022-09-27T16:10:54.5888286Z Ran 1 test in 0.002s
2022-09-27T16:10:54.5888471Z 
2022-09-27T16:10:54.5888583Z OK (skipped=1)
2022-09-27T16:10:54.5888741Z 
2022-09-27T16:10:54.5888869Z Generating XML reports...
2022-09-27T16:10:54.5923002Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161054.xml
2022-09-27T16:10:55.8801267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:55.8801828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:55.8802691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:55.8803177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:55.8837019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:55.8837510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:55.8840717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:55.8841750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:55.9873346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:55.9873843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:55.9875874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:55.9876360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:56.0021877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:56.0022378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:56.0026098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:56.0026586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:56.3888266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:56.3888868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:56.3889630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:56.3989540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:56.3995696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:56.4901769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:56.5235497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:10:56.5236051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:10:56.5236814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:56.5237495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:10:56.5245046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:10:56.5245542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:10:56.6431067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:56.6431843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:56.6432696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:56.6433199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:56.9466644Z ok (4.487s)
2022-09-27T16:10:56.9466868Z 
2022-09-27T16:10:56.9467250Z ----------------------------------------------------------------------
2022-09-27T16:10:56.9467590Z Ran 1 test in 4.488s
2022-09-27T16:10:56.9467763Z 
2022-09-27T16:10:56.9467842Z OK
2022-09-27T16:10:56.9467977Z 
2022-09-27T16:10:56.9468119Z Generating XML reports...
2022-09-27T16:10:56.9503428Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161052.xml
2022-09-27T16:10:56.9823390Z ok (4.328s)
2022-09-27T16:10:56.9823595Z 
2022-09-27T16:10:56.9823920Z ----------------------------------------------------------------------
2022-09-27T16:10:56.9824278Z Ran 1 test in 4.328s
2022-09-27T16:10:56.9824447Z 
2022-09-27T16:10:56.9824545Z OK
2022-09-27T16:10:56.9824680Z 
2022-09-27T16:10:56.9824812Z Generating XML reports...
2022-09-27T16:10:56.9860783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161052.xml
2022-09-27T16:10:57.1793714Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:10:57.1817892Z 
2022-09-27T16:10:57.1818180Z Running tests...
2022-09-27T16:10:57.1818614Z ----------------------------------------------------------------------
2022-09-27T16:10:58.9041111Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:10:58.9233527Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53054
2022-09-27T16:10:58.9239705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53055
2022-09-27T16:10:59.1073628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:59.1074183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:59.1075017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:59.1075507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:59.1779803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:10:59.1780318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:10:59.1782003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:10:59.1782483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:10:59.6582303Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:10:59.6604110Z 
2022-09-27T16:10:59.6604444Z Running tests...
2022-09-27T16:10:59.6604898Z ----------------------------------------------------------------------
2022-09-27T16:10:59.6623989Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:10:59.6624354Z 
2022-09-27T16:10:59.6624680Z ----------------------------------------------------------------------
2022-09-27T16:10:59.6625019Z Ran 1 test in 0.002s
2022-09-27T16:10:59.6625184Z 
2022-09-27T16:10:59.6625292Z OK (skipped=1)
2022-09-27T16:10:59.6625449Z 
2022-09-27T16:10:59.6625558Z Generating XML reports...
2022-09-27T16:10:59.6661708Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161059.xml
2022-09-27T16:10:59.7378461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:10:59.7402325Z 
2022-09-27T16:10:59.7402963Z Running tests...
2022-09-27T16:10:59.7403424Z ----------------------------------------------------------------------
2022-09-27T16:11:00.5417773Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:00.5418789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:00.5419906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:00.5420378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:00.5828958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:00.5829454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:00.5831622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:00.5832103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:01.0733619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:01.1296988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:01.1298460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:01.1340983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:01.1347935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:01.1416624Z STAGE:2022-09-27 16:11:01 53055:53055 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:01.2311126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:01.2377923Z STAGE:2022-09-27 16:11:01 53054:53054 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:01.4052335Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:01.4265816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53166
2022-09-27T16:11:01.4270238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53167
2022-09-27T16:11:01.6216583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:01.6217600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:01.6218753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:01.6219703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:02.1997107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:11:02.2019576Z 
2022-09-27T16:11:02.2020021Z Running tests...
2022-09-27T16:11:02.2020936Z ----------------------------------------------------------------------
2022-09-27T16:11:02.2039443Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:11:02.2040114Z 
2022-09-27T16:11:02.2041030Z ----------------------------------------------------------------------
2022-09-27T16:11:02.2041655Z Ran 1 test in 0.002s
2022-09-27T16:11:02.2041979Z 
2022-09-27T16:11:02.2042184Z OK (skipped=1)
2022-09-27T16:11:02.2042495Z 
2022-09-27T16:11:02.2042728Z Generating XML reports...
2022-09-27T16:11:02.2077485Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161102.xml
2022-09-27T16:11:02.8828398Z [1664295062.882210] [28fd6574525b:53055:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:11:02.8829402Z [1664295062.882210] [28fd6574525b:53055:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:11:02.9428699Z [1664295062.942224] [28fd6574525b:53054:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:11:02.9429889Z [1664295062.942224] [28fd6574525b:53054:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:11:03.0619327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:03.0620329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:03.0621585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:03.0622539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:03.1070178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:03.1071098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:03.1072701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:03.1073589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:03.3614192Z STAGE:2022-09-27 16:11:03 53055:53055 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:11:03.3620142Z STAGE:2022-09-27 16:11:03 53054:53054 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:11:03.6779650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:03.6780676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:03.6781907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:03.6783313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:03.6787809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:03.6790214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:04.1342084Z ok (4.394s)
2022-09-27T16:11:04.1342335Z 
2022-09-27T16:11:04.1342721Z ----------------------------------------------------------------------
2022-09-27T16:11:04.1343070Z Ran 1 test in 4.394s
2022-09-27T16:11:04.1343220Z 
2022-09-27T16:11:04.1343316Z OK
2022-09-27T16:11:04.1343460Z 
2022-09-27T16:11:04.1343597Z Generating XML reports...
2022-09-27T16:11:04.1379811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161059.xml
2022-09-27T16:11:04.2059714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:04.2060198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:04.2062961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:04.2063453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:04.7048287Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:11:04.7070726Z 
2022-09-27T16:11:04.7070878Z Running tests...
2022-09-27T16:11:04.7071624Z ----------------------------------------------------------------------
2022-09-27T16:11:04.7372728Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... ok (7.555s)
2022-09-27T16:11:04.7373056Z 
2022-09-27T16:11:04.7373609Z ----------------------------------------------------------------------
2022-09-27T16:11:04.7373972Z Ran 1 test in 7.555s
2022-09-27T16:11:04.7374141Z 
2022-09-27T16:11:04.7374237Z OK
2022-09-27T16:11:04.7374382Z 
2022-09-27T16:11:04.7374496Z Generating XML reports...
2022-09-27T16:11:04.7409529Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161057.xml
2022-09-27T16:11:06.3536969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:06.3537481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:06.3538719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:06.3539209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:06.4507614Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:06.4678063Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.760s)
2022-09-27T16:11:06.4678950Z 
2022-09-27T16:11:06.4679234Z ----------------------------------------------------------------------
2022-09-27T16:11:06.4679571Z Ran 1 test in 1.761s
2022-09-27T16:11:06.4679736Z 
2022-09-27T16:11:06.4679847Z OK (skipped=1)
2022-09-27T16:11:06.4680005Z 
2022-09-27T16:11:06.4680115Z Generating XML reports...
2022-09-27T16:11:06.4714038Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161104.xml
2022-09-27T16:11:06.8579671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:11:06.8602968Z 
2022-09-27T16:11:06.8603493Z Running tests...
2022-09-27T16:11:06.8604001Z ----------------------------------------------------------------------
2022-09-27T16:11:06.9407845Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:06.9408470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:06.9410735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:06.9411238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:07.4615804Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:11:07.4731248Z 
2022-09-27T16:11:07.4731965Z Running tests...
2022-09-27T16:11:07.4732597Z ----------------------------------------------------------------------
2022-09-27T16:11:07.4791816Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.006s)
2022-09-27T16:11:07.4792549Z 
2022-09-27T16:11:07.4792964Z ----------------------------------------------------------------------
2022-09-27T16:11:07.4793299Z Ran 1 test in 0.006s
2022-09-27T16:11:07.4793581Z 
2022-09-27T16:11:07.4793797Z OK (skipped=1)
2022-09-27T16:11:07.4794007Z 
2022-09-27T16:11:07.4794135Z Generating XML reports...
2022-09-27T16:11:07.4831228Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161107.xml
2022-09-27T16:11:08.5195501Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:08.5393225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53412
2022-09-27T16:11:08.5399516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53413
2022-09-27T16:11:08.5424216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:08.5425189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:08.5428099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:08.5429107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:09.0438577Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:11:09.0460804Z 
2022-09-27T16:11:09.0461270Z Running tests...
2022-09-27T16:11:09.0462122Z ----------------------------------------------------------------------
2022-09-27T16:11:09.5069280Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:09.5069934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:09.5070833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:09.5071328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:10.0058538Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:11:10.0081183Z 
2022-09-27T16:11:10.0081710Z Running tests...
2022-09-27T16:11:10.0082288Z ----------------------------------------------------------------------
2022-09-27T16:11:10.0100680Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:11:10.0101026Z 
2022-09-27T16:11:10.0101315Z ----------------------------------------------------------------------
2022-09-27T16:11:10.0101775Z Ran 1 test in 0.002s
2022-09-27T16:11:10.0102048Z 
2022-09-27T16:11:10.0102163Z OK (skipped=1)
2022-09-27T16:11:10.0102322Z 
2022-09-27T16:11:10.0102451Z Generating XML reports...
2022-09-27T16:11:10.0136140Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161110.xml
2022-09-27T16:11:10.1989187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:10.1989706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:10.1990804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:10.1991292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:10.2298664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:10.2299372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:10.2300969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:10.2301470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:10.6540875Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:10.6742658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53483
2022-09-27T16:11:10.6747923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53484
2022-09-27T16:11:10.7749068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:10.7749649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:10.7750454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:10.7751162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:10.7758049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:10.7758550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:10.7965825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:11:10.7966362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:11:10.7967088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:11:10.7967792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:11:11.3469330Z ok (4.486s)
2022-09-27T16:11:11.3469550Z 
2022-09-27T16:11:11.3469946Z ----------------------------------------------------------------------
2022-09-27T16:11:11.3470269Z Ran 1 test in 4.487s
2022-09-27T16:11:11.3470439Z 
2022-09-27T16:11:11.3470539Z OK
2022-09-27T16:11:11.3470680Z 
2022-09-27T16:11:11.3470819Z Generating XML reports...
2022-09-27T16:11:11.3506245Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161106.xml
2022-09-27T16:11:12.0708802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:12.0709679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:12.0711123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:12.0711617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:12.4033627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:12.4034177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:12.4035696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:12.4036692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:12.4396837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:12.4397805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:12.4399032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:12.4399975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:12.5629926Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:11:12.5653347Z 
2022-09-27T16:11:12.5654045Z Running tests...
2022-09-27T16:11:12.5654971Z ----------------------------------------------------------------------
2022-09-27T16:11:12.5676860Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:11:12.5677982Z 
2022-09-27T16:11:12.5678895Z ----------------------------------------------------------------------
2022-09-27T16:11:12.5679572Z Ran 1 test in 0.002s
2022-09-27T16:11:12.5679900Z 
2022-09-27T16:11:12.5680106Z OK (skipped=1)
2022-09-27T16:11:12.5681040Z 
2022-09-27T16:11:12.5681286Z Generating XML reports...
2022-09-27T16:11:12.5718510Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161112.xml
2022-09-27T16:11:13.5953335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:13.9281858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:13.9321140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:13.9327524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:13.9334223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:14.0360304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:15.8549035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:15.8549665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:15.8552811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:15.8553388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:17.5993761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:11:17.6063967Z 
2022-09-27T16:11:17.6104047Z Running tests...
2022-09-27T16:11:17.6105503Z ----------------------------------------------------------------------
2022-09-27T16:11:18.3333475Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjygkezy
2022-09-27T16:11:18.3335106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjygkezy/_remote_module_non_scriptable.py
2022-09-27T16:11:18.5850562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehnsc6py
2022-09-27T16:11:18.5855609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehnsc6py/_remote_module_non_scriptable.py
2022-09-27T16:11:18.8428197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:18.8428963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:18.8431683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:18.8432450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:20.6217345Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:11:20.6328279Z 
2022-09-27T16:11:20.6328925Z Running tests...
2022-09-27T16:11:20.6329554Z ----------------------------------------------------------------------
2022-09-27T16:11:22.6730028Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:22.7410506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53680
2022-09-27T16:11:22.7421111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53681
2022-09-27T16:11:25.5972773Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:25.6690097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53749
2022-09-27T16:11:25.6696699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53750
2022-09-27T16:11:26.4168927Z ok (17.370s)
2022-09-27T16:11:26.4169221Z 
2022-09-27T16:11:26.4169691Z ----------------------------------------------------------------------
2022-09-27T16:11:26.4170147Z Ran 1 test in 17.371s
2022-09-27T16:11:26.4170355Z 
2022-09-27T16:11:26.4170441Z OK
2022-09-27T16:11:26.4170619Z 
2022-09-27T16:11:26.4170812Z Generating XML reports...
2022-09-27T16:11:26.4289550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161109.xml
2022-09-27T16:11:27.6540157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:27.6540749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:27.6545377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:27.6545906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:27.6604821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:27.6605362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:27.6610704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:27.6611238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:28.2358450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:28.2359424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:28.2360580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:28.2361425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:28.2466843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:28.3414733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:28.3564260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:11:28.3565753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:11:28.3567575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:11:28.3569237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:11:28.5935989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:28.5936581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:28.5938093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:28.5938881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:28.5947177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:28.5947967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:28.5951455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:28.5952262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:28.8356735Z ok (11.229s)
2022-09-27T16:11:28.8357067Z 
2022-09-27T16:11:28.8357827Z ----------------------------------------------------------------------
2022-09-27T16:11:28.8358474Z Ran 1 test in 11.229s
2022-09-27T16:11:28.8358627Z 
2022-09-27T16:11:28.8358731Z OK
2022-09-27T16:11:28.8358868Z 
2022-09-27T16:11:28.8359002Z Generating XML reports...
2022-09-27T16:11:28.8395159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161117.xml
2022-09-27T16:11:29.1307365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:29.1335282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:29.1336678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:29.1408214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:29.1415521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:29.1483013Z STAGE:2022-09-27 16:11:29 53749:53749 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:29.2349974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:29.2521723Z STAGE:2022-09-27 16:11:29 53750:53750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:29.4024014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:29.4024641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:29.4025941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:29.4026502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:29.9305689Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:11:29.9330327Z 
2022-09-27T16:11:29.9330849Z Running tests...
2022-09-27T16:11:29.9331514Z ----------------------------------------------------------------------
2022-09-27T16:11:30.8732503Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:30.8734059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:30.8735267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:30.8736244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:30.9135714Z [1664295090.912972] [28fd6574525b:53750:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:11:30.9136523Z [1664295090.912972] [28fd6574525b:53750:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:11:30.9688390Z [1664295090.968264] [28fd6574525b:53749:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:11:30.9689261Z [1664295090.968264] [28fd6574525b:53749:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:11:31.3701553Z STAGE:2022-09-27 16:11:31 53750:53750 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:11:31.3725513Z STAGE:2022-09-27 16:11:31 53749:53749 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:11:31.3824787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:11:31.3849238Z 
2022-09-27T16:11:31.3849568Z Running tests...
2022-09-27T16:11:31.3850002Z ----------------------------------------------------------------------
2022-09-27T16:11:31.5663576Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:31.5865728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53915
2022-09-27T16:11:31.5871620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53916
2022-09-27T16:11:32.7185938Z ok (12.090s)
2022-09-27T16:11:32.7186157Z 
2022-09-27T16:11:32.7186572Z ----------------------------------------------------------------------
2022-09-27T16:11:32.7186921Z Ran 1 test in 12.090s
2022-09-27T16:11:32.7187092Z 
2022-09-27T16:11:32.7187189Z OK
2022-09-27T16:11:32.7187325Z 
2022-09-27T16:11:32.7187462Z Generating XML reports...
2022-09-27T16:11:32.7221679Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161120.xml
2022-09-27T16:11:33.0144682Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:33.0342920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53980
2022-09-27T16:11:33.0348520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53981
2022-09-27T16:11:33.2299379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:33.2300179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:33.2301361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:33.2301861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:33.2522215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:33.2522709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:33.2525042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:33.2525529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:33.8428218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:33.8624716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:33.8630293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:33.8631728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:33.8636239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:33.9736804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:34.7402944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:34.7403770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:34.7406853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:34.7407647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:34.8339702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:34.8340245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:34.8341276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:34.8341742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:34.9071116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:34.9071605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:34.9074106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:34.9074578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:35.3655653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:35.3656209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:35.3657001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:35.3657687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:35.3764005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:35.4132113Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:11:35.4155722Z 
2022-09-27T16:11:35.4156313Z Running tests...
2022-09-27T16:11:35.4156811Z ----------------------------------------------------------------------
2022-09-27T16:11:35.4670487Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:36.7335630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv23pf5ge
2022-09-27T16:11:36.7336272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv23pf5ge/_remote_module_non_scriptable.py
2022-09-27T16:11:36.8413989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar3ip1k1
2022-09-27T16:11:36.8414595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar3ip1k1/_remote_module_non_scriptable.py
2022-09-27T16:11:37.0942118Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:37.0992075Z ok (7.166s)
2022-09-27T16:11:37.0992471Z 
2022-09-27T16:11:37.0993135Z ----------------------------------------------------------------------
2022-09-27T16:11:37.0993802Z Ran 1 test in 7.166s
2022-09-27T16:11:37.0994092Z 
2022-09-27T16:11:37.0994236Z OK
2022-09-27T16:11:37.0994435Z 
2022-09-27T16:11:37.0994661Z Generating XML reports...
2022-09-27T16:11:37.1030524Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161129.xml
2022-09-27T16:11:37.1144735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54109
2022-09-27T16:11:37.1149705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54110
2022-09-27T16:11:37.9457582Z ok (6.561s)
2022-09-27T16:11:37.9457919Z 
2022-09-27T16:11:37.9458343Z ----------------------------------------------------------------------
2022-09-27T16:11:37.9458668Z Ran 1 test in 6.561s
2022-09-27T16:11:37.9458843Z 
2022-09-27T16:11:37.9458939Z OK
2022-09-27T16:11:37.9459072Z 
2022-09-27T16:11:37.9459203Z Generating XML reports...
2022-09-27T16:11:37.9493960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161131.xml
2022-09-27T16:11:38.8624587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:38.8625592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:38.8626763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:38.8627664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:38.8687042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:38.8687945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:38.8691553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:38.8692527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:39.4310392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:39.4348652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:39.4349482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:39.4412299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:39.4422708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:39.4604904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:39.4605394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:39.4608831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:39.4609532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:39.5362821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:39.9795711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:11:39.9819205Z 
2022-09-27T16:11:39.9819667Z Running tests...
2022-09-27T16:11:39.9820160Z ----------------------------------------------------------------------
2022-09-27T16:11:40.1302266Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:40.1303461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:40.1304657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:40.1305608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:40.6555661Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:11:40.6578307Z 
2022-09-27T16:11:40.6578682Z Running tests...
2022-09-27T16:11:40.6579553Z ----------------------------------------------------------------------
2022-09-27T16:11:40.9276701Z   test_irecv (__main__.TestDistBackendWithSpawn) ... [1664295100.927043] [28fd6574525b:54110:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:11:40.9278030Z [1664295100.927043] [28fd6574525b:54110:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:11:40.9445576Z [1664295100.944108] [28fd6574525b:54109:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:11:40.9446083Z [1664295100.944108] [28fd6574525b:54109:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:11:41.6388651Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:41.6596037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54257
2022-09-27T16:11:41.6601580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54258
2022-09-27T16:11:41.8256247Z ok (6.410s)
2022-09-27T16:11:41.8256701Z 
2022-09-27T16:11:41.8257407Z ----------------------------------------------------------------------
2022-09-27T16:11:41.8257750Z Ran 1 test in 6.410s
2022-09-27T16:11:41.8257921Z 
2022-09-27T16:11:41.8258023Z OK
2022-09-27T16:11:41.8258164Z 
2022-09-27T16:11:41.8258302Z Generating XML reports...
2022-09-27T16:11:41.8292193Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161135.xml
2022-09-27T16:11:42.3159513Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:42.3689921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54323
2022-09-27T16:11:42.3785378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54324
2022-09-27T16:11:43.4775644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:43.4776193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:43.4782209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:43.4783186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:43.4914852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:43.4915362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:43.4918307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:43.4918796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:43.9961385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:43.9961983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:43.9962783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:43.9963491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:43.9969733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:43.9970464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:44.1049181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:44.1049692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:44.1050705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:44.1051187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:44.1052093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:44.1052584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:44.1055021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:44.1055513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:44.1444177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:44.1444657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:44.1447018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:44.1447510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:44.6289097Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:11:44.6312475Z 
2022-09-27T16:11:44.6312682Z Running tests...
2022-09-27T16:11:44.6313259Z ----------------------------------------------------------------------
2022-09-27T16:11:44.6921319Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:44.6922022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:44.6923062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:44.6924085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:44.7031250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:44.7936039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:45.1856039Z ok (4.527s)
2022-09-27T16:11:45.1856445Z 
2022-09-27T16:11:45.1857157Z ----------------------------------------------------------------------
2022-09-27T16:11:45.1857749Z Ran 1 test in 4.528s
2022-09-27T16:11:45.1858048Z 
2022-09-27T16:11:45.1858217Z OK
2022-09-27T16:11:45.1858450Z 
2022-09-27T16:11:45.1858699Z Generating XML reports...
2022-09-27T16:11:45.1896979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161140.xml
2022-09-27T16:11:46.2818396Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:46.3017031Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54445
2022-09-27T16:11:46.3022394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54446
2022-09-27T16:11:47.1724972Z ok (7.190s)
2022-09-27T16:11:47.1725402Z 
2022-09-27T16:11:47.1726176Z ----------------------------------------------------------------------
2022-09-27T16:11:47.1726625Z Ran 1 test in 7.190s
2022-09-27T16:11:47.1726792Z 
2022-09-27T16:11:47.1726868Z OK
2022-09-27T16:11:47.1727002Z 
2022-09-27T16:11:47.1727136Z Generating XML reports...
2022-09-27T16:11:47.1761450Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161139.xml
2022-09-27T16:11:47.4138285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:47.4138890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:47.4140369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:47.4141167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:47.9019676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:11:47.9040678Z 
2022-09-27T16:11:47.9040998Z Running tests...
2022-09-27T16:11:47.9041922Z ----------------------------------------------------------------------
2022-09-27T16:11:47.9517122Z   test_isend (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:47.9517762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:47.9520902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:47.9521678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:48.0083327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:48.0083930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:48.0085541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:48.0086350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:48.4613842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:48.5811213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:48.5812058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:48.5829253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:48.5836755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:48.5903820Z STAGE:2022-09-27 16:11:48 54445:54445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:48.6823934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:48.6888341Z STAGE:2022-09-27 16:11:48 54446:54446 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:49.2750065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:49.2750559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:49.2753104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:49.2753600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:49.5337115Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:49.5540018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54593
2022-09-27T16:11:49.5544160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54594
2022-09-27T16:11:49.7981904Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:11:49.8006466Z 
2022-09-27T16:11:49.8006735Z Running tests...
2022-09-27T16:11:49.8007183Z ----------------------------------------------------------------------
2022-09-27T16:11:50.4085063Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... [1664295110.407988] [28fd6574525b:54445:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:11:50.4098308Z [1664295110.407988] [28fd6574525b:54445:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:11:50.4277830Z [1664295110.427313] [28fd6574525b:54446:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:11:50.4278362Z [1664295110.427313] [28fd6574525b:54446:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:11:50.8168931Z STAGE:2022-09-27 16:11:50 54445:54445 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:11:50.8169575Z STAGE:2022-09-27 16:11:50 54446:54446 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:11:51.1942417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:51.1942948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:51.1944387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:51.1944880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:51.2415601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:51.2416117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:51.2417490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:51.2418113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:51.4627676Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:51.4825268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54666
2022-09-27T16:11:51.4831842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54667
2022-09-27T16:11:51.7636973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:51.7637532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:51.7638329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:51.7639047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:51.7745754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:51.8651266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:52.1164545Z ok (7.485s)
2022-09-27T16:11:52.1165448Z 
2022-09-27T16:11:52.1166885Z ----------------------------------------------------------------------
2022-09-27T16:11:52.1167631Z Ran 1 test in 7.485s
2022-09-27T16:11:52.1167984Z 
2022-09-27T16:11:52.1168173Z OK
2022-09-27T16:11:52.1168419Z 
2022-09-27T16:11:52.1168984Z Generating XML reports...
2022-09-27T16:11:52.1223256Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161144.xml
2022-09-27T16:11:52.3613859Z ok (4.457s)
2022-09-27T16:11:52.3614085Z 
2022-09-27T16:11:52.3614497Z ----------------------------------------------------------------------
2022-09-27T16:11:52.3614824Z Ran 1 test in 4.457s
2022-09-27T16:11:52.3615001Z 
2022-09-27T16:11:52.3615098Z OK
2022-09-27T16:11:52.3615234Z 
2022-09-27T16:11:52.3615375Z Generating XML reports...
2022-09-27T16:11:52.3650355Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161147.xml
2022-09-27T16:11:53.1091803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:53.1092309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:53.1093561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:53.1094039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:53.1358614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:53.1359089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:53.1362779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:53.1363257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:53.6692225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:53.7142516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:53.7143345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:53.7198690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:53.7205643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:53.8156601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:54.3590037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:54.3590538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:54.3593203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:54.3593707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:54.5201383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:54.5201873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:54.5204096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:54.5204598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:54.8673588Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:11:54.8697564Z 
2022-09-27T16:11:54.8697820Z Running tests...
2022-09-27T16:11:54.8698251Z ----------------------------------------------------------------------
2022-09-27T16:11:55.0307892Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:11:55.0329472Z 
2022-09-27T16:11:55.0329728Z Running tests...
2022-09-27T16:11:55.0330160Z ----------------------------------------------------------------------
2022-09-27T16:11:55.0832556Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4svy56m
2022-09-27T16:11:55.0833504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4svy56m/_remote_module_non_scriptable.py
2022-09-27T16:11:55.2108240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputahet9j
2022-09-27T16:11:55.2109624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputahet9j/_remote_module_non_scriptable.py
2022-09-27T16:11:55.7309197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:55.7310312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:55.7326204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:55.7327885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:11:56.3951952Z ok (6.594s)
2022-09-27T16:11:56.3952339Z 
2022-09-27T16:11:56.3953035Z ----------------------------------------------------------------------
2022-09-27T16:11:56.3953658Z Ran 1 test in 6.594s
2022-09-27T16:11:56.3953982Z 
2022-09-27T16:11:56.3954183Z OK
2022-09-27T16:11:56.3954433Z 
2022-09-27T16:11:56.3954709Z Generating XML reports...
2022-09-27T16:11:56.3990043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161149.xml
2022-09-27T16:11:56.5275638Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:56.5475584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54828
2022-09-27T16:11:56.5481407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54829
2022-09-27T16:11:56.7272424Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:11:56.7470266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54831
2022-09-27T16:11:56.7476151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54832
2022-09-27T16:11:58.3414738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:58.3415640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:58.3417092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:58.3417601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:58.3418182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:58.3418651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:58.3422512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:58.3423016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:58.5429882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:58.5430590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:58.5432042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:58.5432551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:58.5525953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:58.5526432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:58.5530324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:58.5530826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:58.6303376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:11:58.6304226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:11:58.6307485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:11:58.6307993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:11:58.8591319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:58.8605499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:58.8606641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:58.8692198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:58.8698877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:58.8764047Z STAGE:2022-09-27 16:11:58 54829:54829 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:58.9617133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:58.9681370Z STAGE:2022-09-27 16:11:58 54828:54828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:59.0973824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:11:59.0974397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:11:59.0975175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:59.0975889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:11:59.0983227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:11:59.0983737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:11:59.1083178Z STAGE:2022-09-27 16:11:59 54832:54832 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:11:59 54831:54831 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:11:59.1083604Z 
2022-09-27T16:11:59.1121681Z STAGE:2022-09-27 16:11:59 54831:54831 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:11:59.1124616Z STAGE:2022-09-27 16:11:59 54832:54832 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:11:59.1243555Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:11:59.1266094Z 
2022-09-27T16:11:59.1266528Z Running tests...
2022-09-27T16:11:59.1267050Z ----------------------------------------------------------------------
2022-09-27T16:11:59.6548417Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... ok (4.621s)
2022-09-27T16:11:59.6549015Z 
2022-09-27T16:11:59.6549421Z ----------------------------------------------------------------------
2022-09-27T16:11:59.6549785Z Ran 1 test in 4.622s
2022-09-27T16:11:59.6549954Z 
2022-09-27T16:11:59.6550031Z OK
2022-09-27T16:11:59.6550168Z 
2022-09-27T16:11:59.6550298Z Generating XML reports...
2022-09-27T16:11:59.6587628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161155.xml
2022-09-27T16:12:00.6451753Z [1664295120.644668] [28fd6574525b:54829:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:12:00.6452340Z [1664295120.644668] [28fd6574525b:54829:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:12:00.6590104Z [1664295120.658662] [28fd6574525b:54828:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:12:00.6590842Z [1664295120.658662] [28fd6574525b:54828:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:12:00.7756163Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:00.7953890Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55023
2022-09-27T16:12:00.7959418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55024
2022-09-27T16:12:01.0860845Z STAGE:2022-09-27 16:12:01 54828:54828 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:12:01.0862093Z STAGE:2022-09-27 16:12:01 54829:54829 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:12:01.9740639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:01.9741382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:01.9742627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:01.9743115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:02.3616158Z ok (7.491s)
2022-09-27T16:12:02.3616773Z 
2022-09-27T16:12:02.3617163Z ----------------------------------------------------------------------
2022-09-27T16:12:02.3617763Z Ran 1 test in 7.492s
2022-09-27T16:12:02.3617940Z 
2022-09-27T16:12:02.3618033Z OK
2022-09-27T16:12:02.3618168Z 
2022-09-27T16:12:02.3618285Z Generating XML reports...
2022-09-27T16:12:02.3751408Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161154.xml
2022-09-27T16:12:02.4857683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:12:02.4880986Z 
2022-09-27T16:12:02.4881206Z Running tests...
2022-09-27T16:12:02.4882101Z ----------------------------------------------------------------------
2022-09-27T16:12:02.5397404Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:02.5397999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:02.5399272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:02.5400099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:02.5625852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:02.5626458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:02.5627714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:02.5628516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:03.0640415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:03.0934271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:03.0935131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:03.0944536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:03.0951962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:03.1948913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:04.2624193Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:04.2826706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55164
2022-09-27T16:12:04.2831824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55165
2022-09-27T16:12:04.4992592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5oqkwwqv
2022-09-27T16:12:04.4993877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5oqkwwqv/_remote_module_non_scriptable.py
2022-09-27T16:12:04.5439274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:04.5439980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:04.5441655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:04.5442133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:04.5660410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5eqnjhc8
2022-09-27T16:12:04.5662561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5eqnjhc8/_remote_module_non_scriptable.py
2022-09-27T16:12:05.1275476Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:12:05.1297879Z 
2022-09-27T16:12:05.1298357Z Running tests...
2022-09-27T16:12:05.1299148Z ----------------------------------------------------------------------
2022-09-27T16:12:05.1316296Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-09-27T16:12:05.1316637Z 
2022-09-27T16:12:05.1317173Z ----------------------------------------------------------------------
2022-09-27T16:12:05.1317547Z Ran 1 test in 0.002s
2022-09-27T16:12:05.1317718Z 
2022-09-27T16:12:05.1317832Z OK (skipped=1)
2022-09-27T16:12:05.1318000Z 
2022-09-27T16:12:05.1318137Z Generating XML reports...
2022-09-27T16:12:05.1352321Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161205.xml
2022-09-27T16:12:05.1918809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:05.1919355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:05.1937135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:05.1937658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:05.8069951Z ok (6.680s)
2022-09-27T16:12:05.8070315Z 
2022-09-27T16:12:05.8070991Z ----------------------------------------------------------------------
2022-09-27T16:12:05.8071636Z Ran 1 test in 6.680s
2022-09-27T16:12:05.8071947Z 
2022-09-27T16:12:05.8072125Z OK
2022-09-27T16:12:05.8072387Z 
2022-09-27T16:12:05.8072607Z Generating XML reports...
2022-09-27T16:12:05.8107346Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161159.xml
2022-09-27T16:12:05.9470634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:05.9471627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:05.9472754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:05.9473723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:05.9643090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:05.9644066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:05.9646548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:05.9647548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:06.5896109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:06.5897400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:06.5898228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:06.5898958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:06.6004608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:06.6069472Z STAGE:2022-09-27 16:12:06 55165:55165 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:12:06.6908528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:06.6973331Z STAGE:2022-09-27 16:12:06 55164:55164 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:12:06.7011249Z STAGE:2022-09-27 16:12:06 55165:55165 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:12:06.8003840Z STAGE:2022-09-27 16:12:06 55164:55164 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:12:07.1808640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:07.1809466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:07.1811945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:07.1812460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:07.1897917Z ok (4.702s)
2022-09-27T16:12:07.1898089Z 
2022-09-27T16:12:07.1898389Z ----------------------------------------------------------------------
2022-09-27T16:12:07.1898716Z Ran 1 test in 4.702s
2022-09-27T16:12:07.1898882Z 
2022-09-27T16:12:07.1898978Z OK
2022-09-27T16:12:07.1899116Z 
2022-09-27T16:12:07.1899617Z Generating XML reports...
2022-09-27T16:12:07.1936044Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161202.xml
2022-09-27T16:12:07.6909579Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:12:07.6932659Z 
2022-09-27T16:12:07.6932944Z Running tests...
2022-09-27T16:12:07.6933370Z ----------------------------------------------------------------------
2022-09-27T16:12:07.6951357Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-09-27T16:12:07.6952772Z 
2022-09-27T16:12:07.6953287Z ----------------------------------------------------------------------
2022-09-27T16:12:07.6953631Z Ran 1 test in 0.002s
2022-09-27T16:12:07.6953802Z 
2022-09-27T16:12:07.6953895Z OK (skipped=1)
2022-09-27T16:12:07.6954059Z 
2022-09-27T16:12:07.6954191Z Generating XML reports...
2022-09-27T16:12:07.6988970Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161207.xml
2022-09-27T16:12:07.9221001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:07.9221520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:07.9224206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:07.9224703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:08.4770644Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:08.4793096Z 
2022-09-27T16:12:08.4793549Z Running tests...
2022-09-27T16:12:08.4794047Z ----------------------------------------------------------------------
2022-09-27T16:12:09.3183711Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:09.3184474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:09.3185317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:09.3185818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:09.6188942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:09.6189879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:09.6191391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:09.6191988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:09.8171518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:12:09.8192920Z 
2022-09-27T16:12:09.8193352Z Running tests...
2022-09-27T16:12:09.8193873Z ----------------------------------------------------------------------
2022-09-27T16:12:10.1147079Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:10.1148054Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:12:10.1171988Z 
2022-09-27T16:12:10.1172405Z Running tests...
2022-09-27T16:12:10.1172858Z ----------------------------------------------------------------------
2022-09-27T16:12:10.1350958Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55389
2022-09-27T16:12:10.1357048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55390
2022-09-27T16:12:11.4558588Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:11.4764231Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55455
2022-09-27T16:12:11.4769406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55456
2022-09-27T16:12:11.7657174Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:11.7858941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55458
2022-09-27T16:12:11.7864762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55459
2022-09-27T16:12:11.8087673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:11.8088164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:11.8090867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:11.8091374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:11.8408248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:11.8408746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:11.8410499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:11.8411009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:12.3763597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:12.4074277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:12.4075842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:12.4168629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:12.4175574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:12.5132732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:12.9428314Z skip: Need at least 4 CUDA devices (4.463s)
2022-09-27T16:12:12.9428567Z 
2022-09-27T16:12:12.9428956Z ----------------------------------------------------------------------
2022-09-27T16:12:12.9429281Z Ran 1 test in 4.463s
2022-09-27T16:12:12.9429448Z 
2022-09-27T16:12:12.9429559Z OK (skipped=1)
2022-09-27T16:12:12.9429723Z 
2022-09-27T16:12:12.9429850Z Generating XML reports...
2022-09-27T16:12:12.9464441Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161208.xml
2022-09-27T16:12:13.2226329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:13.2227096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:13.2228919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:13.2229412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:13.2632736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:13.2633452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:13.2635611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:13.2636114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:13.5628743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:13.5629261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:13.5630131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:13.5630624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:13.5656937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:13.5657415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:13.5661316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:13.5661816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:13.7970558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:13.7971105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:13.7971903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:13.7972614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:13.7981886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:13.7983637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:13.7985958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:12:13.7988272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:12:13.7988956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:12:13.8089397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:12:13.8198241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:12:13.8198812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:12:13.8199557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:12:13.8200565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:12:14.1028231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:14.1357321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:14.1359270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:14.1434306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:14.1442878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:14.2366276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:15.1036802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:15.1037325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:15.1038716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:15.1039190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:15.5471722Z [1664295135.546539] [28fd6574525b:55458:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:12:15.5472944Z [1664295135.546539] [28fd6574525b:55458:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:12:15.5547341Z [1664295135.554388] [28fd6574525b:55459:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:12:15.5547857Z [1664295135.554388] [28fd6574525b:55459:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:12:15.6704445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:15.6727881Z 
2022-09-27T16:12:15.6728339Z Running tests...
2022-09-27T16:12:15.6728823Z ----------------------------------------------------------------------
2022-09-27T16:12:16.5626938Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:12:16.9975594Z ok (6.880s)
2022-09-27T16:12:16.9975861Z 
2022-09-27T16:12:16.9976285Z ----------------------------------------------------------------------
2022-09-27T16:12:16.9976632Z Ran 1 test in 6.880s
2022-09-27T16:12:16.9976803Z 
2022-09-27T16:12:16.9976912Z OK
2022-09-27T16:12:16.9977039Z 
2022-09-27T16:12:16.9977178Z Generating XML reports...
2022-09-27T16:12:17.0011834Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161210.xml
2022-09-27T16:12:17.3627446Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:17.3830019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55674
2022-09-27T16:12:17.3835913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55675
2022-09-27T16:12:18.9747316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:18.9747848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:18.9749566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:18.9750301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:19.0120852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:19.0121717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:19.0124650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:19.0125406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:19.0208294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:19.0209034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:19.0212270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:19.0213037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:19.4985898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:19.5177007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:19.5177823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:19.5188150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:19.5194423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:19.5290858Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:12:19.5315836Z 
2022-09-27T16:12:19.5316119Z Running tests...
2022-09-27T16:12:19.5316564Z ----------------------------------------------------------------------
2022-09-27T16:12:19.6191830Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:19.9898628Z skip: Need at least 4 CUDA devices (4.317s)
2022-09-27T16:12:19.9898882Z 
2022-09-27T16:12:19.9899275Z ----------------------------------------------------------------------
2022-09-27T16:12:19.9899616Z Ran 1 test in 4.317s
2022-09-27T16:12:19.9899784Z 
2022-09-27T16:12:19.9899877Z OK (skipped=1)
2022-09-27T16:12:19.9900043Z 
2022-09-27T16:12:19.9903116Z Generating XML reports...
2022-09-27T16:12:19.9935850Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161215.xml
2022-09-27T16:12:21.1799086Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:21.2004158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55813
2022-09-27T16:12:21.2010563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55814
2022-09-27T16:12:22.1150692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:22.1151676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:22.1152573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:22.1153065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:22.5930799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:22.5953980Z 
2022-09-27T16:12:22.5954648Z Running tests...
2022-09-27T16:12:22.5955148Z ----------------------------------------------------------------------
2022-09-27T16:12:22.8482602Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:22.8483269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:22.8483901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:22.8484373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:22.8709384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:22.8709887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:22.8711620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:22.8712083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:23.3657868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:23.3921127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:23.3922193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:23.3961972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:23.3969733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:23.4070652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtf3hv3d
2022-09-27T16:12:23.4073413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtf3hv3d/_remote_module_non_scriptable.py
2022-09-27T16:12:23.4935231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:23.5038347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0ttkk1w
2022-09-27T16:12:23.5040903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0ttkk1w/_remote_module_non_scriptable.py
2022-09-27T16:12:24.2400704Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:24.2603346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55886
2022-09-27T16:12:24.2608242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55887
2022-09-27T16:12:24.8923585Z [1664295144.891493] [28fd6574525b:55813:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:12:24.9481681Z [1664295144.891493] [28fd6574525b:55813:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:12:24.9483246Z [1664295144.947333] [28fd6574525b:55814:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:12:24.9485117Z [1664295144.947333] [28fd6574525b:55814:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:12:25.8109573Z ok (6.279s)
2022-09-27T16:12:25.8109829Z 
2022-09-27T16:12:25.8110251Z ----------------------------------------------------------------------
2022-09-27T16:12:25.8110595Z Ran 1 test in 6.279s
2022-09-27T16:12:25.8110765Z 
2022-09-27T16:12:25.8110862Z OK
2022-09-27T16:12:25.8110980Z 
2022-09-27T16:12:25.8111116Z Generating XML reports...
2022-09-27T16:12:25.8145842Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161219.xml
2022-09-27T16:12:25.9254404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:25.9255171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:25.9256329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:25.9257207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:25.9719885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:25.9720788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:25.9723221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:25.9723986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:26.4479397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:26.4882194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:26.4883257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:26.4884326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:26.4891834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:26.4892738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:27.7929349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:27.7929882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:27.7930944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:27.7931519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:27.7979039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lswrdp4
2022-09-27T16:12:27.7981957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lswrdp4/_remote_module_non_scriptable.py
2022-09-27T16:12:28.2767406Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:12:28.2789632Z 
2022-09-27T16:12:28.2789858Z Running tests...
2022-09-27T16:12:28.2790439Z ----------------------------------------------------------------------
2022-09-27T16:12:28.2812539Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl & Gloo backend support DistributedDataParallel (0.002s)
2022-09-27T16:12:28.2813147Z 
2022-09-27T16:12:28.2813448Z ----------------------------------------------------------------------
2022-09-27T16:12:28.2813789Z Ran 1 test in 0.002s
2022-09-27T16:12:28.2813994Z 
2022-09-27T16:12:28.2814200Z OK (skipped=1)
2022-09-27T16:12:28.2814475Z 
2022-09-27T16:12:28.2814590Z Generating XML reports...
2022-09-27T16:12:28.2848558Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161228.xml
2022-09-27T16:12:30.1282645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:30.1283492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:30.1285428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:30.1286188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:30.6260186Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:12:30.6282982Z 
2022-09-27T16:12:30.6283528Z Running tests...
2022-09-27T16:12:30.6284266Z ----------------------------------------------------------------------
2022-09-27T16:12:32.1972885Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:32.2162418Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56039
2022-09-27T16:12:32.2169136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56040
2022-09-27T16:12:33.0187749Z ok (23.199s)
2022-09-27T16:12:33.0188274Z 
2022-09-27T16:12:33.0188947Z ----------------------------------------------------------------------
2022-09-27T16:12:33.0189607Z Ran 1 test in 23.199s
2022-09-27T16:12:33.0189941Z 
2022-09-27T16:12:33.0190155Z OK
2022-09-27T16:12:33.0190408Z 
2022-09-27T16:12:33.0190697Z Generating XML reports...
2022-09-27T16:12:33.0228988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161209.xml
2022-09-27T16:12:33.7144088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvyivctz
2022-09-27T16:12:33.7145317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvyivctz/_remote_module_non_scriptable.py
2022-09-27T16:12:33.8662879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:33.8663946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:33.8665191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:33.8666628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:33.8954409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:33.8955435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:33.8956639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:33.8957620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:34.2247746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:34.2256917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:12:34.4150141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:34.4356857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:34.4357714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:34.4457115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:34.4463560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:34.5369755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:34.8831591Z ok (12.287s)
2022-09-27T16:12:34.8831930Z 
2022-09-27T16:12:34.8832353Z ----------------------------------------------------------------------
2022-09-27T16:12:34.8832684Z Ran 1 test in 12.288s
2022-09-27T16:12:34.8832859Z 
2022-09-27T16:12:34.8832955Z OK
2022-09-27T16:12:34.8833091Z 
2022-09-27T16:12:34.8833232Z Generating XML reports...
2022-09-27T16:12:34.8881847Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161222.xml
2022-09-27T16:12:35.2084955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:35.2085492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:35.2086891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:35.2087381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:35.7382685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:12:35.7407522Z 
2022-09-27T16:12:35.7408364Z Running tests...
2022-09-27T16:12:35.7408923Z ----------------------------------------------------------------------
2022-09-27T16:12:35.8674480Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_fkm7dq
2022-09-27T16:12:35.8675665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_fkm7dq/_remote_module_non_scriptable.py
2022-09-27T16:12:35.9911626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeimass6l
2022-09-27T16:12:35.9912530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeimass6l/_remote_module_non_scriptable.py
2022-09-27T16:12:36.0178860Z [1664295156.017357] [28fd6574525b:56039:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:12:36.0179418Z [1664295156.017357] [28fd6574525b:56039:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:12:36.0253483Z [1664295156.024999] [28fd6574525b:56040:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:12:36.0254250Z [1664295156.024999] [28fd6574525b:56040:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:12:36.8480433Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:12:36.9033152Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:12:37.0626188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:37.0626695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:37.0628188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:37.0628681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:37.4241376Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:12:37.4439290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56199
2022-09-27T16:12:37.4446248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56200
2022-09-27T16:12:37.5286898Z ok (6.900s)
2022-09-27T16:12:37.5287239Z 
2022-09-27T16:12:37.5287674Z ----------------------------------------------------------------------
2022-09-27T16:12:37.5288021Z Ran 1 test in 6.900s
2022-09-27T16:12:37.5288171Z 
2022-09-27T16:12:37.5288270Z OK
2022-09-27T16:12:37.5288406Z 
2022-09-27T16:12:37.5288543Z Generating XML reports...
2022-09-27T16:12:37.5322558Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161230.xml
2022-09-27T16:12:37.5805748Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:37.5828210Z 
2022-09-27T16:12:37.5828607Z Running tests...
2022-09-27T16:12:37.5849380Z ----------------------------------------------------------------------
2022-09-27T16:12:37.5851010Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:37.5851846Z 
2022-09-27T16:12:37.5852517Z ----------------------------------------------------------------------
2022-09-27T16:12:37.5853295Z Ran 1 test in 0.002s
2022-09-27T16:12:37.5853675Z 
2022-09-27T16:12:37.5853928Z OK (skipped=1)
2022-09-27T16:12:37.5854310Z 
2022-09-27T16:12:37.5854589Z Generating XML reports...
2022-09-27T16:12:37.5886363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161237.xml
2022-09-27T16:12:39.1171520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:39.1172028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:39.1174320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:39.1174812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:39.1639741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:39.1640836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:39.1643346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:39.1643824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:39.6325969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:39.6326475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:39.6329622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:39.6330116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:39.6754488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:39.6754976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:39.6758663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:39.6759141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:39.6946378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:12:39.6947015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:12:39.6947708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:39.6948405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:12:39.7056994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:12:39.7060178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:12:39.7960751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:12:39.7964035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:12:39.7964854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:12:39.7974550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:12:39.8173361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:12:39.8174181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:12:39.8174902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:12:39.8175608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:12:40.1386710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:40.1411357Z 
2022-09-27T16:12:40.1411866Z Running tests...
2022-09-27T16:12:40.1412323Z ----------------------------------------------------------------------
2022-09-27T16:12:40.1432196Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:40.1432774Z 
2022-09-27T16:12:40.1433374Z ----------------------------------------------------------------------
2022-09-27T16:12:40.1433775Z Ran 1 test in 0.002s
2022-09-27T16:12:40.1433950Z 
2022-09-27T16:12:40.1434066Z OK (skipped=1)
2022-09-27T16:12:40.1434223Z 
2022-09-27T16:12:40.1434350Z Generating XML reports...
2022-09-27T16:12:40.1470391Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161240.xml
2022-09-27T16:12:40.1723230Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:12:40.1748332Z 
2022-09-27T16:12:40.1748713Z Running tests...
2022-09-27T16:12:40.1749176Z ----------------------------------------------------------------------
2022-09-27T16:12:40.1767536Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:12:40.1769232Z 
2022-09-27T16:12:40.1769896Z ----------------------------------------------------------------------
2022-09-27T16:12:40.1770241Z Ran 1 test in 0.002s
2022-09-27T16:12:40.1770433Z 
2022-09-27T16:12:40.1770548Z OK (skipped=1)
2022-09-27T16:12:40.1770688Z 
2022-09-27T16:12:40.1770815Z Generating XML reports...
2022-09-27T16:12:40.1806435Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161240.xml
2022-09-27T16:12:42.1158969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:42.1159494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:42.1161958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:42.1162450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:42.1608517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:42.1609016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:42.1611673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:42.1612152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:42.4209674Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:12:42.4210207Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:12:42.6550585Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:12:42.6574064Z 
2022-09-27T16:12:42.6574402Z Running tests...
2022-09-27T16:12:42.6575069Z ----------------------------------------------------------------------
2022-09-27T16:12:42.6576707Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:42.6593077Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:12:42.6593482Z 
2022-09-27T16:12:42.6593860Z ----------------------------------------------------------------------
2022-09-27T16:12:42.6594380Z Ran 1 test in 0.002s
2022-09-27T16:12:42.6594551Z 
2022-09-27T16:12:42.6594664Z OK (skipped=1)
2022-09-27T16:12:42.6594826Z 
2022-09-27T16:12:42.6594954Z Generating XML reports...
2022-09-27T16:12:42.6602615Z 
2022-09-27T16:12:42.6602930Z Running tests...
2022-09-27T16:12:42.6603396Z ----------------------------------------------------------------------
2022-09-27T16:12:42.6623099Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:42.6624212Z 
2022-09-27T16:12:42.6624620Z ----------------------------------------------------------------------
2022-09-27T16:12:42.6625336Z Ran 1 test in 0.002s
2022-09-27T16:12:42.6625667Z 
2022-09-27T16:12:42.6625866Z OK (skipped=1)
2022-09-27T16:12:42.6626087Z 
2022-09-27T16:12:42.6626225Z Generating XML reports...
2022-09-27T16:12:42.6628653Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161242.xml
2022-09-27T16:12:42.6659716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161242.xml
2022-09-27T16:12:44.5607264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:44.5607805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:44.5608806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:44.5609304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:45.0584284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:45.0606442Z 
2022-09-27T16:12:45.0606765Z Running tests...
2022-09-27T16:12:45.0607181Z ----------------------------------------------------------------------
2022-09-27T16:12:45.0626430Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:45.0626761Z 
2022-09-27T16:12:45.0627037Z ----------------------------------------------------------------------
2022-09-27T16:12:45.0627382Z Ran 1 test in 0.002s
2022-09-27T16:12:45.0627546Z 
2022-09-27T16:12:45.0627644Z OK (skipped=1)
2022-09-27T16:12:45.0627805Z 
2022-09-27T16:12:45.0627933Z Generating XML reports...
2022-09-27T16:12:45.0661261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161245.xml
2022-09-27T16:12:46.9263013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:46.9263590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:46.9265361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:46.9265858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:47.4164370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:47.4187464Z 
2022-09-27T16:12:47.4187879Z Running tests...
2022-09-27T16:12:47.4188599Z ----------------------------------------------------------------------
2022-09-27T16:12:47.4207426Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:47.4207984Z 
2022-09-27T16:12:47.4208500Z ----------------------------------------------------------------------
2022-09-27T16:12:47.4208868Z Ran 1 test in 0.002s
2022-09-27T16:12:47.4209042Z 
2022-09-27T16:12:47.4209154Z OK (skipped=1)
2022-09-27T16:12:47.4209297Z 
2022-09-27T16:12:47.4209523Z Generating XML reports...
2022-09-27T16:12:47.4242002Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161247.xml
2022-09-27T16:12:49.2579368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:49.2580089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:49.2581715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:49.2582210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:49.7527722Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:49.7550231Z 
2022-09-27T16:12:49.7550385Z Running tests...
2022-09-27T16:12:49.7551297Z ----------------------------------------------------------------------
2022-09-27T16:12:49.7570502Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:49.7571180Z 
2022-09-27T16:12:49.7571502Z ----------------------------------------------------------------------
2022-09-27T16:12:49.7571860Z Ran 1 test in 0.002s
2022-09-27T16:12:49.7572514Z 
2022-09-27T16:12:49.7572640Z OK (skipped=1)
2022-09-27T16:12:49.7572802Z 
2022-09-27T16:12:49.7572934Z Generating XML reports...
2022-09-27T16:12:49.7605681Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161249.xml
2022-09-27T16:12:51.6010687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:51.6011210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:51.6012728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:51.6013373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:52.1035026Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:52.1058071Z 
2022-09-27T16:12:52.1058262Z Running tests...
2022-09-27T16:12:52.1058976Z ----------------------------------------------------------------------
2022-09-27T16:12:52.1079114Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:52.1079452Z 
2022-09-27T16:12:52.1079745Z ----------------------------------------------------------------------
2022-09-27T16:12:52.1080574Z Ran 1 test in 0.002s
2022-09-27T16:12:52.1080753Z 
2022-09-27T16:12:52.1080868Z OK (skipped=1)
2022-09-27T16:12:52.1081032Z 
2022-09-27T16:12:52.1081162Z Generating XML reports...
2022-09-27T16:12:52.1114750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161252.xml
2022-09-27T16:12:53.9574142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:53.9574660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:53.9576294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:53.9576949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:54.4450732Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:54.4473204Z 
2022-09-27T16:12:54.4473678Z Running tests...
2022-09-27T16:12:54.4474324Z ----------------------------------------------------------------------
2022-09-27T16:12:54.4493725Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:54.4494243Z 
2022-09-27T16:12:54.4494539Z ----------------------------------------------------------------------
2022-09-27T16:12:54.4494873Z Ran 1 test in 0.002s
2022-09-27T16:12:54.4495322Z 
2022-09-27T16:12:54.4495557Z OK (skipped=1)
2022-09-27T16:12:54.4495746Z 
2022-09-27T16:12:54.4495877Z Generating XML reports...
2022-09-27T16:12:54.4528223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161254.xml
2022-09-27T16:12:56.2777743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:56.2778248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:56.2779943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:56.2780438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:56.7827196Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:56.7850794Z 
2022-09-27T16:12:56.7851202Z Running tests...
2022-09-27T16:12:56.7851689Z ----------------------------------------------------------------------
2022-09-27T16:12:56.7870142Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:56.7870945Z 
2022-09-27T16:12:56.7871242Z ----------------------------------------------------------------------
2022-09-27T16:12:56.7871580Z Ran 1 test in 0.002s
2022-09-27T16:12:56.7871730Z 
2022-09-27T16:12:56.7871840Z OK (skipped=1)
2022-09-27T16:12:56.7871996Z 
2022-09-27T16:12:56.7872120Z Generating XML reports...
2022-09-27T16:12:56.7905727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161256.xml
2022-09-27T16:12:58.5825951Z ok (22.842s)
2022-09-27T16:12:58.5826167Z 
2022-09-27T16:12:58.5826799Z ----------------------------------------------------------------------
2022-09-27T16:12:58.5827163Z Ran 1 test in 22.842s
2022-09-27T16:12:58.5827313Z 
2022-09-27T16:12:58.5827418Z OK
2022-09-27T16:12:58.5827555Z 
2022-09-27T16:12:58.5827767Z Generating XML reports...
2022-09-27T16:12:58.5861679Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161235.xml
2022-09-27T16:12:58.6429691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:12:58.6430385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:12:58.6432148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:12:58.6432722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:12:59.1511083Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:12:59.1534338Z 
2022-09-27T16:12:59.1534633Z Running tests...
2022-09-27T16:12:59.1535085Z ----------------------------------------------------------------------
2022-09-27T16:12:59.1554449Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:12:59.1554772Z 
2022-09-27T16:12:59.1555061Z ----------------------------------------------------------------------
2022-09-27T16:12:59.1555390Z Ran 1 test in 0.002s
2022-09-27T16:12:59.1555553Z 
2022-09-27T16:12:59.1555669Z OK (skipped=1)
2022-09-27T16:12:59.1555825Z 
2022-09-27T16:12:59.1555946Z Generating XML reports...
2022-09-27T16:12:59.1590726Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161259.xml
2022-09-27T16:13:00.6508970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:00.6509504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:00.6511629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:00.6512411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:01.0622218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:01.0622790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:01.0623808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:01.0624300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:01.1356789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:01.1379780Z 
2022-09-27T16:13:01.1380054Z Running tests...
2022-09-27T16:13:01.1380657Z ----------------------------------------------------------------------
2022-09-27T16:13:01.5734746Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:01.5757522Z 
2022-09-27T16:13:01.5757817Z Running tests...
2022-09-27T16:13:01.5758512Z ----------------------------------------------------------------------
2022-09-27T16:13:02.7791188Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:02.7987245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56723
2022-09-27T16:13:02.7993190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56724
2022-09-27T16:13:03.1609249Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:03.1810345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56726
2022-09-27T16:13:03.1816704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56727
2022-09-27T16:13:04.4590555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:04.4591565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:04.4592735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:04.4593713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:04.5232613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:04.5233603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:04.5234739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:04.5235696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:04.8602933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:04.8603899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:04.8605093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:04.8606050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:04.9124172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:04.9125178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:04.9126970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:04.9127977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:05.0470631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:05.0471207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:05.0472258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:05.0472982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:05.0579158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:05.1486067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:05.3739598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:05.4287656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:05.4288547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:05.4347282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:05.4354084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:05.5058751Z skip: Skipped due to small world size. (4.367s)
2022-09-27T16:13:05.5059187Z 
2022-09-27T16:13:05.5059840Z ----------------------------------------------------------------------
2022-09-27T16:13:05.5060452Z Ran 1 test in 4.368s
2022-09-27T16:13:05.5060759Z 
2022-09-27T16:13:05.5060965Z OK (skipped=1)
2022-09-27T16:13:05.5061277Z 
2022-09-27T16:13:05.5061495Z Generating XML reports...
2022-09-27T16:13:05.5098359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161301.xml
2022-09-27T16:13:05.5302046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:07.0673210Z STAGE:2022-09-27 16:13:07 56727:56727 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:07.2028058Z STAGE:2022-09-27 16:13:07 56726:56726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:07.2884679Z STAGE:2022-09-27 16:13:07 56726:56726 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:07.2885347Z STAGE:2022-09-27 16:13:07 56727:56727 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:07.4578044Z STAGE:2022-09-27 16:13:07 56727:56727 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:07.4592959Z STAGE:2022-09-27 16:13:07 56727:56727 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:07.4593543Z STAGE:2022-09-27 16:13:07 56726:56726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:07.4609480Z STAGE:2022-09-27 16:13:07 56726:56726 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:07.6333428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:07.6333971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:07.6336456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:07.6336969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:07.9921040Z ok (6.416s)
2022-09-27T16:13:07.9921636Z 
2022-09-27T16:13:07.9922032Z ----------------------------------------------------------------------
2022-09-27T16:13:07.9922377Z Ran 1 test in 6.416s
2022-09-27T16:13:07.9922542Z 
2022-09-27T16:13:07.9922623Z OK
2022-09-27T16:13:07.9922764Z 
2022-09-27T16:13:07.9924442Z Generating XML reports...
2022-09-27T16:13:07.9957715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161301.xml
2022-09-27T16:13:08.1384052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:08.1408083Z 
2022-09-27T16:13:08.1408846Z Running tests...
2022-09-27T16:13:08.1409397Z ----------------------------------------------------------------------
2022-09-27T16:13:09.7863082Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:09.8060268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56950
2022-09-27T16:13:09.8067076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56951
2022-09-27T16:13:09.9625614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:09.9626622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:09.9627807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:09.9628760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:10.4649498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:10.4692698Z 
2022-09-27T16:13:10.4693403Z Running tests...
2022-09-27T16:13:10.4694957Z ----------------------------------------------------------------------
2022-09-27T16:13:10.4739560Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.004s)
2022-09-27T16:13:10.4741614Z 
2022-09-27T16:13:10.4744648Z ----------------------------------------------------------------------
2022-09-27T16:13:10.4745370Z Ran 1 test in 0.005s
2022-09-27T16:13:10.4746283Z 
2022-09-27T16:13:10.4748292Z OK (skipped=1)
2022-09-27T16:13:10.4749933Z 
2022-09-27T16:13:10.4751806Z Generating XML reports...
2022-09-27T16:13:10.4836650Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161310.xml
2022-09-27T16:13:11.4388666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:11.4392861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:11.4397652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:11.4399304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:11.4822702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:11.4829952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:11.4831210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:11.4832150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:12.0315594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:12.0316513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:12.0317938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:12.0319368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:12.0323160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:12.0325889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:12.4311979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:12.4312485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:12.4314909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:12.4315642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:12.9204659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:12.9227603Z 
2022-09-27T16:13:12.9228042Z Running tests...
2022-09-27T16:13:12.9228532Z ----------------------------------------------------------------------
2022-09-27T16:13:14.0450296Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:13:14.4807316Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:14.5008210Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57060
2022-09-27T16:13:14.5014617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57061
2022-09-27T16:13:14.5164333Z ok (6.375s)
2022-09-27T16:13:14.5164598Z 
2022-09-27T16:13:14.5165113Z ----------------------------------------------------------------------
2022-09-27T16:13:14.5165517Z Ran 1 test in 6.376s
2022-09-27T16:13:14.5165693Z 
2022-09-27T16:13:14.5165791Z OK
2022-09-27T16:13:14.5165928Z 
2022-09-27T16:13:14.5166044Z Generating XML reports...
2022-09-27T16:13:14.5202717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161308.xml
2022-09-27T16:13:16.0623802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:16.0624770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:16.0625928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:16.0626886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:16.1077918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:16.1078922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:16.1080834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:16.1081804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:16.5629688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:16.5980039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:16.5980527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:16.5982932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:16.5983433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:16.6065327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:16.6066051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:16.6136789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:16.6144458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:16.7079081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:17.1027582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:17.1050430Z 
2022-09-27T16:13:17.1050781Z Running tests...
2022-09-27T16:13:17.1051615Z ----------------------------------------------------------------------
2022-09-27T16:13:18.7439214Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:18.7640005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57172
2022-09-27T16:13:18.7645233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57173
2022-09-27T16:13:19.7144546Z ok (6.791s)
2022-09-27T16:13:19.7144776Z 
2022-09-27T16:13:19.7145181Z ----------------------------------------------------------------------
2022-09-27T16:13:19.7145508Z Ran 1 test in 6.792s
2022-09-27T16:13:19.7145674Z 
2022-09-27T16:13:19.7145779Z OK
2022-09-27T16:13:19.7145924Z 
2022-09-27T16:13:19.7146058Z Generating XML reports...
2022-09-27T16:13:19.7181222Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161312.xml
2022-09-27T16:13:20.4089458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:20.4089976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:20.4092368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:20.4092870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:20.4266390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:20.4266858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:20.4270445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:20.4270948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:21.0106228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:21.0106834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:21.0107646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:21.0108354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:21.0214198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:21.1117051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:21.1325571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:13:21.1326557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:13:21.1327917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:13:21.1328892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:13:21.1329433Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-09-27T16:13:21.1329854Z No ranks successfully processed in monitoredBarrier.
2022-09-27T16:13:21.1359176Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-09-27T16:13:21.5711885Z ok (4.466s)
2022-09-27T16:13:21.5712128Z 
2022-09-27T16:13:21.5712495Z ----------------------------------------------------------------------
2022-09-27T16:13:21.5712843Z Ran 1 test in 4.466s
2022-09-27T16:13:21.5713014Z 
2022-09-27T16:13:21.5713107Z OK
2022-09-27T16:13:21.5713247Z 
2022-09-27T16:13:21.5713386Z Generating XML reports...
2022-09-27T16:13:21.5749304Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161317.xml
2022-09-27T16:13:21.7466827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:21.7467650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:21.7469521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:21.7470017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:22.2379906Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:22.2401267Z 
2022-09-27T16:13:22.2401739Z Running tests...
2022-09-27T16:13:22.2402265Z ----------------------------------------------------------------------
2022-09-27T16:13:22.2422119Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:13:22.2422429Z 
2022-09-27T16:13:22.2422723Z ----------------------------------------------------------------------
2022-09-27T16:13:22.2423044Z Ran 1 test in 0.002s
2022-09-27T16:13:22.2423225Z 
2022-09-27T16:13:22.2423337Z OK (skipped=1)
2022-09-27T16:13:22.2423501Z 
2022-09-27T16:13:22.2423644Z Generating XML reports...
2022-09-27T16:13:22.2457815Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161322.xml
2022-09-27T16:13:23.6717571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:23.6718111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:23.6720721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:23.6721210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:24.1558288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:24.1581039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:24.1581549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:24.1581776Z 
2022-09-27T16:13:24.1581890Z Running tests...
2022-09-27T16:13:24.1582287Z ----------------------------------------------------------------------
2022-09-27T16:13:24.1584982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:24.1585481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:24.6505764Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:24.6527849Z 
2022-09-27T16:13:24.6528331Z Running tests...
2022-09-27T16:13:24.6528832Z ----------------------------------------------------------------------
2022-09-27T16:13:25.7387722Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:25.7586618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57359
2022-09-27T16:13:25.7592211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57360
2022-09-27T16:13:26.1849640Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:26.2042330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57362
2022-09-27T16:13:26.2048771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57363
2022-09-27T16:13:27.4450648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:27.4451176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:27.4452923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:27.4453401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:27.4632397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:27.4632916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:27.4635953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:27.4636447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:27.8663415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:27.8663917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:27.8665734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:27.8666210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:27.8828962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:27.8829461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:27.8833075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:27.8833807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:27.9887775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:27.9888338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:27.9889113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:27.9889800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:27.9995913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:28.0899351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:28.1010523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:13:28.1011062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:13:28.1011778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:13:28.1012484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:13:28.2118182Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:13:28.3756974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:28.3965280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:28.3966118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:28.4062575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:28.4069402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:28.4979926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:28.6668887Z ok (4.508s)
2022-09-27T16:13:28.6669097Z 
2022-09-27T16:13:28.6669503Z ----------------------------------------------------------------------
2022-09-27T16:13:28.6669827Z Ran 1 test in 4.509s
2022-09-27T16:13:28.6669997Z 
2022-09-27T16:13:28.6670094Z OK
2022-09-27T16:13:28.6670234Z 
2022-09-27T16:13:28.6670372Z Generating XML reports...
2022-09-27T16:13:28.6706424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161324.xml
2022-09-27T16:13:30.0188906Z STAGE:2022-09-27 16:13:30 57362:57362 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:30.1459784Z STAGE:2022-09-27 16:13:30 57363:57363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:30.2157846Z STAGE:2022-09-27 16:13:30 57363:57363 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:30.2158460Z STAGE:2022-09-27 16:13:30 57362:57362 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:30.3826855Z STAGE:2022-09-27 16:13:30 57362:57362 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:30.3841230Z STAGE:2022-09-27 16:13:30 57362:57362 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:30.3857557Z STAGE:2022-09-27 16:13:30 57363:57363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:30.3873115Z STAGE:2022-09-27 16:13:30 57363:57363 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:30.7617387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:30.7618363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:30.7619204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:30.7619698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:31.0153592Z ok (6.362s)
2022-09-27T16:13:31.0153815Z 
2022-09-27T16:13:31.0154238Z ----------------------------------------------------------------------
2022-09-27T16:13:31.0154565Z Ran 1 test in 6.362s
2022-09-27T16:13:31.0154734Z 
2022-09-27T16:13:31.0154829Z OK
2022-09-27T16:13:31.0154966Z 
2022-09-27T16:13:31.0155100Z Generating XML reports...
2022-09-27T16:13:31.0190251Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161324.xml
2022-09-27T16:13:31.2528432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:31.2550438Z 
2022-09-27T16:13:31.2550731Z Running tests...
2022-09-27T16:13:31.2551161Z ----------------------------------------------------------------------
2022-09-27T16:13:32.8254799Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:32.8443709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57592
2022-09-27T16:13:32.8449945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57593
2022-09-27T16:13:33.0118884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:33.0119393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:33.0122403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:33.0122901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:33.5636680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:33.5676706Z 
2022-09-27T16:13:33.5677148Z Running tests...
2022-09-27T16:13:33.5677660Z ----------------------------------------------------------------------
2022-09-27T16:13:34.4509443Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:34.4510122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:34.4511207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:34.4512269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:34.4523741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:34.4524266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:34.4527695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:34.4528469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:34.9899522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:34.9900090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:34.9900868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:34.9901592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:34.9907961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:34.9909208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:35.1327898Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:35.1535640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57669
2022-09-27T16:13:35.1540187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57670
2022-09-27T16:13:35.3510973Z skip: Skipped due to small world size. (4.096s)
2022-09-27T16:13:35.3511263Z 
2022-09-27T16:13:35.3511646Z ----------------------------------------------------------------------
2022-09-27T16:13:35.3511975Z Ran 1 test in 4.096s
2022-09-27T16:13:35.3512141Z 
2022-09-27T16:13:35.3512253Z OK (skipped=1)
2022-09-27T16:13:35.3512409Z 
2022-09-27T16:13:35.3512536Z Generating XML reports...
2022-09-27T16:13:35.3548798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161331.xml
2022-09-27T16:13:36.8082822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:36.8083824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:36.8085008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:36.8085538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:36.8999354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:36.8999865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:36.9001530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:36.9002044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:37.3066981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:37.3945251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:37.3946073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:37.3978587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:37.3985821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:37.4819924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:37.4820452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:37.4822992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:37.4823519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:37.4957495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:38.0027018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:38.0050954Z 
2022-09-27T16:13:38.0051118Z Running tests...
2022-09-27T16:13:38.0051944Z ----------------------------------------------------------------------
2022-09-27T16:13:38.0076482Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:13:38.0077049Z 
2022-09-27T16:13:38.0077824Z ----------------------------------------------------------------------
2022-09-27T16:13:38.0078512Z Ran 1 test in 0.003s
2022-09-27T16:13:38.0078867Z 
2022-09-27T16:13:38.0079081Z OK (skipped=1)
2022-09-27T16:13:38.0079393Z 
2022-09-27T16:13:38.0079646Z Generating XML reports...
2022-09-27T16:13:38.0115189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161337.xml
2022-09-27T16:13:38.9795308Z STAGE:2022-09-27 16:13:38 57670:57670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:39.0416115Z STAGE:2022-09-27 16:13:39 57669:57669 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:39.1571483Z STAGE:2022-09-27 16:13:39 57669:57669 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:39.1572771Z STAGE:2022-09-27 16:13:39 57670:57670 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:39.3258639Z STAGE:2022-09-27 16:13:39 57670:57670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:39.3274793Z STAGE:2022-09-27 16:13:39 57670:57670 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:39.3325170Z STAGE:2022-09-27 16:13:39 57669:57669 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:39.3343106Z STAGE:2022-09-27 16:13:39 57669:57669 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:39.9654490Z ok (6.397s)
2022-09-27T16:13:39.9654751Z 
2022-09-27T16:13:39.9655138Z ----------------------------------------------------------------------
2022-09-27T16:13:39.9655476Z Ran 1 test in 6.398s
2022-09-27T16:13:39.9655646Z 
2022-09-27T16:13:39.9655746Z OK
2022-09-27T16:13:39.9655881Z 
2022-09-27T16:13:39.9656000Z Generating XML reports...
2022-09-27T16:13:39.9690519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161333.xml
2022-09-27T16:13:39.9789710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:39.9790195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:39.9793798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:39.9794299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:40.4870245Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:40.4892983Z 
2022-09-27T16:13:40.4893252Z Running tests...
2022-09-27T16:13:40.4893682Z ----------------------------------------------------------------------
2022-09-27T16:13:40.4919298Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:13:40.4919634Z 
2022-09-27T16:13:40.4919898Z ----------------------------------------------------------------------
2022-09-27T16:13:40.4920572Z Ran 1 test in 0.003s
2022-09-27T16:13:40.4920864Z 
2022-09-27T16:13:40.4921074Z OK (skipped=1)
2022-09-27T16:13:40.4921698Z 
2022-09-27T16:13:40.4921858Z Generating XML reports...
2022-09-27T16:13:40.4955541Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161340.xml
2022-09-27T16:13:42.0843759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:42.0844277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:42.0846754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:42.0847254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:42.4192284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:42.4192801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:42.4195384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:42.4195888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:42.5762536Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:42.5786057Z 
2022-09-27T16:13:42.5786201Z Running tests...
2022-09-27T16:13:42.5786912Z ----------------------------------------------------------------------
2022-09-27T16:13:42.5807146Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:13:42.5807474Z 
2022-09-27T16:13:42.5807765Z ----------------------------------------------------------------------
2022-09-27T16:13:42.5808098Z Ran 1 test in 0.002s
2022-09-27T16:13:42.5808264Z 
2022-09-27T16:13:42.5808374Z OK (skipped=1)
2022-09-27T16:13:42.5808516Z 
2022-09-27T16:13:42.5808647Z Generating XML reports...
2022-09-27T16:13:42.5843830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161342.xml
2022-09-27T16:13:42.8966025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:42.8988495Z 
2022-09-27T16:13:42.8988646Z Running tests...
2022-09-27T16:13:42.8989350Z ----------------------------------------------------------------------
2022-09-27T16:13:42.9012177Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:13:42.9012520Z 
2022-09-27T16:13:42.9012772Z ----------------------------------------------------------------------
2022-09-27T16:13:42.9013110Z Ran 1 test in 0.002s
2022-09-27T16:13:42.9013279Z 
2022-09-27T16:13:42.9013388Z OK (skipped=1)
2022-09-27T16:13:42.9013549Z 
2022-09-27T16:13:42.9013675Z Generating XML reports...
2022-09-27T16:13:42.9047773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161342.xml
2022-09-27T16:13:44.5319523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:44.5320076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:44.5325748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:44.5326283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:44.8245149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:44.8245645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:44.8248444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:44.8248923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:45.0251225Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:45.0274522Z 
2022-09-27T16:13:45.0274901Z Running tests...
2022-09-27T16:13:45.0275343Z ----------------------------------------------------------------------
2022-09-27T16:13:45.0293956Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:13:45.0294952Z 
2022-09-27T16:13:45.0295347Z ----------------------------------------------------------------------
2022-09-27T16:13:45.0295688Z Ran 1 test in 0.002s
2022-09-27T16:13:45.0295856Z 
2022-09-27T16:13:45.0295965Z OK (skipped=1)
2022-09-27T16:13:45.0296124Z 
2022-09-27T16:13:45.0296235Z Generating XML reports...
2022-09-27T16:13:45.0330886Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161345.xml
2022-09-27T16:13:45.3041915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:45.3064760Z 
2022-09-27T16:13:45.3064911Z Running tests...
2022-09-27T16:13:45.3065342Z ----------------------------------------------------------------------
2022-09-27T16:13:45.3093412Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-09-27T16:13:45.3094005Z 
2022-09-27T16:13:45.3094285Z ----------------------------------------------------------------------
2022-09-27T16:13:45.3094606Z Ran 1 test in 0.003s
2022-09-27T16:13:45.3094774Z 
2022-09-27T16:13:45.3094893Z OK (skipped=1)
2022-09-27T16:13:45.3095048Z 
2022-09-27T16:13:45.3095176Z Generating XML reports...
2022-09-27T16:13:45.3130561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161345.xml
2022-09-27T16:13:46.9855418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:46.9855973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:46.9857991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:46.9858492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:47.1619922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:47.1620422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:47.1622710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:47.1623210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:47.4801335Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:47.4825338Z 
2022-09-27T16:13:47.4825664Z Running tests...
2022-09-27T16:13:47.4826109Z ----------------------------------------------------------------------
2022-09-27T16:13:47.4850368Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:13:47.4850856Z 
2022-09-27T16:13:47.4851316Z ----------------------------------------------------------------------
2022-09-27T16:13:47.4851696Z Ran 1 test in 0.003s
2022-09-27T16:13:47.4851869Z 
2022-09-27T16:13:47.4851995Z OK (skipped=1)
2022-09-27T16:13:47.4852159Z 
2022-09-27T16:13:47.4852269Z Generating XML reports...
2022-09-27T16:13:47.4887011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161347.xml
2022-09-27T16:13:47.6435117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:47.6458403Z 
2022-09-27T16:13:47.6458539Z Running tests...
2022-09-27T16:13:47.6459142Z ----------------------------------------------------------------------
2022-09-27T16:13:47.6482939Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-09-27T16:13:47.6483380Z 
2022-09-27T16:13:47.6483669Z ----------------------------------------------------------------------
2022-09-27T16:13:47.6484032Z Ran 1 test in 0.002s
2022-09-27T16:13:47.6484201Z 
2022-09-27T16:13:47.6484315Z OK (skipped=1)
2022-09-27T16:13:47.6484455Z 
2022-09-27T16:13:47.6484583Z Generating XML reports...
2022-09-27T16:13:47.6519120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161347.xml
2022-09-27T16:13:49.4167975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:49.4168713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:49.4170723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:49.4171520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:49.6083511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:49.6084331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:49.6086900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:49.6087661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:49.9020594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:49.9043939Z 
2022-09-27T16:13:49.9044352Z Running tests...
2022-09-27T16:13:49.9045014Z ----------------------------------------------------------------------
2022-09-27T16:13:49.9063299Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:13:49.9063723Z 
2022-09-27T16:13:49.9063991Z ----------------------------------------------------------------------
2022-09-27T16:13:49.9064335Z Ran 1 test in 0.002s
2022-09-27T16:13:49.9064626Z 
2022-09-27T16:13:49.9064834Z OK (skipped=1)
2022-09-27T16:13:49.9065015Z 
2022-09-27T16:13:49.9065144Z Generating XML reports...
2022-09-27T16:13:49.9099915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161349.xml
2022-09-27T16:13:50.1054386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:50.1077947Z 
2022-09-27T16:13:50.1078331Z Running tests...
2022-09-27T16:13:50.1079009Z ----------------------------------------------------------------------
2022-09-27T16:13:50.1100464Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:13:50.1100775Z 
2022-09-27T16:13:50.1101186Z ----------------------------------------------------------------------
2022-09-27T16:13:50.1101650Z Ran 1 test in 0.002s
2022-09-27T16:13:50.1101818Z 
2022-09-27T16:13:50.1101940Z OK (skipped=1)
2022-09-27T16:13:50.1102079Z 
2022-09-27T16:13:50.1102217Z Generating XML reports...
2022-09-27T16:13:50.1137686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161350.xml
2022-09-27T16:13:51.8033216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:51.8034223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:51.8036453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:51.8037425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:51.9743050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:51.9744427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:51.9745679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:51.9746666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:52.2993799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:13:52.3018829Z 
2022-09-27T16:13:52.3019256Z Running tests...
2022-09-27T16:13:52.3020152Z ----------------------------------------------------------------------
2022-09-27T16:13:52.4542012Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:52.4564431Z 
2022-09-27T16:13:52.4564921Z Running tests...
2022-09-27T16:13:52.4565409Z ----------------------------------------------------------------------
2022-09-27T16:13:52.4588846Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:13:52.4589178Z 
2022-09-27T16:13:52.4589741Z ----------------------------------------------------------------------
2022-09-27T16:13:52.4590418Z Ran 1 test in 0.003s
2022-09-27T16:13:52.4590569Z 
2022-09-27T16:13:52.4590692Z OK (skipped=1)
2022-09-27T16:13:52.4590849Z 
2022-09-27T16:13:52.4590978Z Generating XML reports...
2022-09-27T16:13:52.4624984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161352.xml
2022-09-27T16:13:53.9412826Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:53.9609327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58184
2022-09-27T16:13:53.9616255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58185
2022-09-27T16:13:54.3431570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:54.3432070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:54.3433368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:54.3433871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:54.8869722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:54.8892361Z 
2022-09-27T16:13:54.8892508Z Running tests...
2022-09-27T16:13:54.8893541Z ----------------------------------------------------------------------
2022-09-27T16:13:54.8914168Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:13:54.8914534Z 
2022-09-27T16:13:54.8914875Z ----------------------------------------------------------------------
2022-09-27T16:13:54.8915217Z Ran 1 test in 0.002s
2022-09-27T16:13:54.8915414Z 
2022-09-27T16:13:54.8915536Z OK (skipped=1)
2022-09-27T16:13:54.8915689Z 
2022-09-27T16:13:54.8915799Z Generating XML reports...
2022-09-27T16:13:54.8950270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161354.xml
2022-09-27T16:13:55.5156173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:55.5157185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:55.5158331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:55.5159282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:55.5691192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:55.5692117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:55.5694632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:55.5695620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:56.0184894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:13:56.1457466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:13:56.1458367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:56.1502686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:13:56.1510733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:13:56.2471563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:13:56.7581428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:13:56.7582871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:13:56.7583991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:13:56.7584874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:13:57.2640729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:13:57.2664188Z 
2022-09-27T16:13:57.2664695Z Running tests...
2022-09-27T16:13:57.2665193Z ----------------------------------------------------------------------
2022-09-27T16:13:57.7445628Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:13:57 58184:58184 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:57.8162113Z STAGE:2022-09-27 16:13:57 58185:58185 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:58.8746163Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:13:58.8941288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58302
2022-09-27T16:13:58.8947253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58303
2022-09-27T16:13:59.2106784Z STAGE:2022-09-27 16:13:59 58184:58184 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:59.2136810Z STAGE:2022-09-27 16:13:59 58185:58185 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:59.3800014Z STAGE:2022-09-27 16:13:59 58184:58184 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:59.3909412Z STAGE:2022-09-27 16:13:59 58185:58185 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:13:59.3953659Z STAGE:2022-09-27 16:13:59 58185:58185 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:13:59.3980379Z STAGE:2022-09-27 16:13:59 58184:58184 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:00.1747605Z ok (7.873s)
2022-09-27T16:14:00.1747835Z 
2022-09-27T16:14:00.1748238Z ----------------------------------------------------------------------
2022-09-27T16:14:00.1748589Z Ran 1 test in 7.873s
2022-09-27T16:14:00.1748740Z 
2022-09-27T16:14:00.1748843Z OK
2022-09-27T16:14:00.1748979Z 
2022-09-27T16:14:00.1749114Z Generating XML reports...
2022-09-27T16:14:00.1783862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161352.xml
2022-09-27T16:14:00.5182193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:00.5182699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:00.5185076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:00.5185588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:00.5590523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:00.5591020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:00.5593391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:00.5593864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:01.0973260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:01.0973847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:01.0974659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:01.0975371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:01.1081374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:01.2002733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:01.7014340Z ok (4.435s)
2022-09-27T16:14:01.7014608Z 
2022-09-27T16:14:01.7015256Z ----------------------------------------------------------------------
2022-09-27T16:14:01.7015627Z Ran 1 test in 4.435s
2022-09-27T16:14:01.7015802Z 
2022-09-27T16:14:01.7015900Z OK
2022-09-27T16:14:01.7016046Z 
2022-09-27T16:14:01.7016182Z Generating XML reports...
2022-09-27T16:14:01.7050920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161357.xml
2022-09-27T16:14:02.2304730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:02.2305702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:02.2307224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:02.2307717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:02.7354293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:02.7394488Z 
2022-09-27T16:14:02.7396865Z Running tests...
2022-09-27T16:14:02.7399895Z ----------------------------------------------------------------------
2022-09-27T16:14:03.7678083Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:03.7678703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:03.7680100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:03.7680895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:04.2947775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:04.2971235Z 
2022-09-27T16:14:04.2971626Z Running tests...
2022-09-27T16:14:04.2972127Z ----------------------------------------------------------------------
2022-09-27T16:14:04.3317371Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:04.3511765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58453
2022-09-27T16:14:04.3517792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58454
2022-09-27T16:14:05.9177816Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:05.9370663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58519
2022-09-27T16:14:05.9376339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58520
2022-09-27T16:14:05.9665963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:05.9666447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:05.9669334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:05.9669834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:06.0794762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:06.0795251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:06.0797045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:06.0797541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:06.4691719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:06.6475935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:06.6489842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:06.6524867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:06.6605097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:06.7517200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:07.6094754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:07.6095287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:07.6097744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:07.6098487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:07.6100545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:07.6101016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:07.6104661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:07.6105131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:08.1742109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:08.1742687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:08.1743490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:08.1744197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:08.1851249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:08.2605011Z STAGE:2022-09-27 16:14:08 58454:58454 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:14:08.2757840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:08.3049815Z STAGE:2022-09-27 16:14:08 58453:58453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:14:08.6452371Z ok (4.348s)
2022-09-27T16:14:08.6452593Z 
2022-09-27T16:14:08.6453304Z ----------------------------------------------------------------------
2022-09-27T16:14:08.6453652Z Ran 1 test in 4.348s
2022-09-27T16:14:08.6453824Z 
2022-09-27T16:14:08.6453925Z OK
2022-09-27T16:14:08.6454074Z 
2022-09-27T16:14:08.6454207Z Generating XML reports...
2022-09-27T16:14:08.6490682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161404.xml
2022-09-27T16:14:09.6740507Z STAGE:2022-09-27 16:14:09 58453:58453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:09.6771034Z STAGE:2022-09-27 16:14:09 58454:58454 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:09.9007474Z STAGE:2022-09-27 16:14:09 58453:58453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:14:09.9076434Z STAGE:2022-09-27 16:14:09 58454:58454 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:14:09.9374106Z STAGE:2022-09-27 16:14:09 58453:58453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:09.9579684Z STAGE:2022-09-27 16:14:09 58454:58454 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:10.6657552Z ok (7.926s)
2022-09-27T16:14:10.6657814Z 
2022-09-27T16:14:10.6658207Z ----------------------------------------------------------------------
2022-09-27T16:14:10.6658548Z Ran 1 test in 7.927s
2022-09-27T16:14:10.6658717Z 
2022-09-27T16:14:10.6658812Z OK
2022-09-27T16:14:10.6658931Z 
2022-09-27T16:14:10.6659065Z Generating XML reports...
2022-09-27T16:14:10.6694252Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161402.xml
2022-09-27T16:14:10.8250431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:10.8250929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:10.8253210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:10.8253688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:11.3409341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:11.3432440Z 
2022-09-27T16:14:11.3432714Z Running tests...
2022-09-27T16:14:11.3433157Z ----------------------------------------------------------------------
2022-09-27T16:14:11.3453164Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:14:11.3453488Z 
2022-09-27T16:14:11.3453769Z ----------------------------------------------------------------------
2022-09-27T16:14:11.3454079Z Ran 1 test in 0.002s
2022-09-27T16:14:11.3454245Z 
2022-09-27T16:14:11.3454359Z OK (skipped=1)
2022-09-27T16:14:11.3454515Z 
2022-09-27T16:14:11.3454640Z Generating XML reports...
2022-09-27T16:14:11.3489390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161411.xml
2022-09-27T16:14:12.6991166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:12.6991728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:12.6992963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:12.6993437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:13.1928221Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:13.1950987Z 
2022-09-27T16:14:13.1951427Z Running tests...
2022-09-27T16:14:13.1951952Z ----------------------------------------------------------------------
2022-09-27T16:14:13.1971157Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:14:13.1971518Z 
2022-09-27T16:14:13.1971808Z ----------------------------------------------------------------------
2022-09-27T16:14:13.1972126Z Ran 1 test in 0.002s
2022-09-27T16:14:13.1972302Z 
2022-09-27T16:14:13.1972412Z OK (skipped=1)
2022-09-27T16:14:13.1972571Z 
2022-09-27T16:14:13.1972695Z Generating XML reports...
2022-09-27T16:14:13.2007373Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161413.xml
2022-09-27T16:14:13.2548855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:13.2549357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:13.2550913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:13.2551401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:13.7422621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:13.7444402Z 
2022-09-27T16:14:13.7444694Z Running tests...
2022-09-27T16:14:13.7445127Z ----------------------------------------------------------------------
2022-09-27T16:14:13.7464449Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:14:13.7464816Z 
2022-09-27T16:14:13.7465110Z ----------------------------------------------------------------------
2022-09-27T16:14:13.7465449Z Ran 1 test in 0.002s
2022-09-27T16:14:13.7465619Z 
2022-09-27T16:14:13.7465730Z OK (skipped=1)
2022-09-27T16:14:13.7465888Z 
2022-09-27T16:14:13.7467848Z Generating XML reports...
2022-09-27T16:14:13.7499794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161413.xml
2022-09-27T16:14:15.2061350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:15.2062069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:15.2063396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:15.2063906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:15.7115639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:15.7139134Z 
2022-09-27T16:14:15.7139551Z Running tests...
2022-09-27T16:14:15.7140045Z ----------------------------------------------------------------------
2022-09-27T16:14:15.7158559Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:14:15.7158898Z 
2022-09-27T16:14:15.7159199Z ----------------------------------------------------------------------
2022-09-27T16:14:15.7159540Z Ran 1 test in 0.002s
2022-09-27T16:14:15.7159724Z 
2022-09-27T16:14:15.7159836Z OK (skipped=1)
2022-09-27T16:14:15.7159998Z 
2022-09-27T16:14:15.7160126Z Generating XML reports...
2022-09-27T16:14:15.7174005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:15.7174537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:15.7178571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:15.7179065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:15.7194637Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161415.xml
2022-09-27T16:14:16.2159113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:16.2181631Z 
2022-09-27T16:14:16.2182038Z Running tests...
2022-09-27T16:14:16.2183142Z ----------------------------------------------------------------------
2022-09-27T16:14:17.6031330Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:17.6032058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:17.6034261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:17.6034754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:17.7920999Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:17.8086748Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.590s)
2022-09-27T16:14:17.8087320Z 
2022-09-27T16:14:17.8087594Z ----------------------------------------------------------------------
2022-09-27T16:14:17.8087928Z Ran 1 test in 1.590s
2022-09-27T16:14:17.8088320Z 
2022-09-27T16:14:17.8088433Z OK (skipped=1)
2022-09-27T16:14:17.8088572Z 
2022-09-27T16:14:17.8088699Z Generating XML reports...
2022-09-27T16:14:17.8121410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161416.xml
2022-09-27T16:14:18.1165259Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:18.1188202Z 
2022-09-27T16:14:18.1188622Z Running tests...
2022-09-27T16:14:18.1189150Z ----------------------------------------------------------------------
2022-09-27T16:14:18.1211138Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:14:18.1211492Z 
2022-09-27T16:14:18.1212000Z ----------------------------------------------------------------------
2022-09-27T16:14:18.1212352Z Ran 1 test in 0.002s
2022-09-27T16:14:18.1212520Z 
2022-09-27T16:14:18.1212632Z OK (skipped=1)
2022-09-27T16:14:18.1212821Z 
2022-09-27T16:14:18.1213056Z Generating XML reports...
2022-09-27T16:14:18.1247333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161418.xml
2022-09-27T16:14:19.8470208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:19.8470727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:19.8471866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:19.8472339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:20.0470683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:20.0471190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:20.0473183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:20.0473666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:20.3285619Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:20.3308365Z 
2022-09-27T16:14:20.3308627Z Running tests...
2022-09-27T16:14:20.3309061Z ----------------------------------------------------------------------
2022-09-27T16:14:20.5377391Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:20.5400332Z 
2022-09-27T16:14:20.5400676Z Running tests...
2022-09-27T16:14:20.5401471Z ----------------------------------------------------------------------
2022-09-27T16:14:20.5419147Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:14:20.5419477Z 
2022-09-27T16:14:20.5419760Z ----------------------------------------------------------------------
2022-09-27T16:14:20.5420074Z Ran 1 test in 0.002s
2022-09-27T16:14:20.5420240Z 
2022-09-27T16:14:20.5420351Z OK (skipped=1)
2022-09-27T16:14:20.5420512Z 
2022-09-27T16:14:20.5420637Z Generating XML reports...
2022-09-27T16:14:20.5455223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161420.xml
2022-09-27T16:14:21.9214991Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:21.9407255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58919
2022-09-27T16:14:21.9413391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58920
2022-09-27T16:14:22.4669001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:22.4669518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:22.4670873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:22.4671342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:23.0530803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:23.0553926Z 
2022-09-27T16:14:23.0554333Z Running tests...
2022-09-27T16:14:23.0554844Z ----------------------------------------------------------------------
2022-09-27T16:14:23.0571450Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-09-27T16:14:23.0572496Z 
2022-09-27T16:14:23.0572822Z ----------------------------------------------------------------------
2022-09-27T16:14:23.0573161Z Ran 1 test in 0.002s
2022-09-27T16:14:23.0573452Z 
2022-09-27T16:14:23.0573652Z OK (skipped=1)
2022-09-27T16:14:23.0573856Z 
2022-09-27T16:14:23.0573989Z Generating XML reports...
2022-09-27T16:14:23.0607375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161423.xml
2022-09-27T16:14:23.5794661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:23.5795179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:23.5797357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:23.5797844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:23.6032163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:23.6032656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:23.6035967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:23.6036453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:24.1536798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:24.1537497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:24.1538307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:24.1539001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:24.1644900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:24.2550395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:24.8893010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:24.8893569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:24.8896029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:24.8896524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:25.4018498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:25.4042306Z 
2022-09-27T16:14:25.4042725Z Running tests...
2022-09-27T16:14:25.4043561Z ----------------------------------------------------------------------
2022-09-27T16:14:25.4063239Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-09-27T16:14:25.4063956Z 
2022-09-27T16:14:25.4064509Z ----------------------------------------------------------------------
2022-09-27T16:14:25.4065118Z Ran 1 test in 0.002s
2022-09-27T16:14:25.4065858Z 
2022-09-27T16:14:25.4066045Z OK (skipped=1)
2022-09-27T16:14:25.4066355Z 
2022-09-27T16:14:25.4066589Z Generating XML reports...
2022-09-27T16:14:25.4102327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161425.xml
2022-09-27T16:14:25.4748339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcu81rdh_
2022-09-27T16:14:25.4749324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcu81rdh_/_remote_module_non_scriptable.py
2022-09-27T16:14:25.5394875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2x_v6a2
2022-09-27T16:14:25.5395860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2x_v6a2/_remote_module_non_scriptable.py
2022-09-27T16:14:26.5526529Z ok (6.221s)
2022-09-27T16:14:26.5526894Z 
2022-09-27T16:14:26.5527638Z ----------------------------------------------------------------------
2022-09-27T16:14:26.5528302Z Ran 1 test in 6.222s
2022-09-27T16:14:26.5528473Z 
2022-09-27T16:14:26.5528568Z OK
2022-09-27T16:14:26.5528707Z 
2022-09-27T16:14:26.5528827Z Generating XML reports...
2022-09-27T16:14:26.5562549Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161420.xml
2022-09-27T16:14:27.3286360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:27.3287053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:27.3288167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:27.3288844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:27.8760864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:27.8784183Z 
2022-09-27T16:14:27.8784567Z Running tests...
2022-09-27T16:14:27.8785000Z ----------------------------------------------------------------------
2022-09-27T16:14:27.8803806Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-09-27T16:14:27.8804166Z 
2022-09-27T16:14:27.8804456Z ----------------------------------------------------------------------
2022-09-27T16:14:27.8804791Z Ran 1 test in 0.002s
2022-09-27T16:14:27.8804938Z 
2022-09-27T16:14:27.8805050Z OK (skipped=1)
2022-09-27T16:14:27.8805204Z 
2022-09-27T16:14:27.8805330Z Generating XML reports...
2022-09-27T16:14:27.8838552Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161427.xml
2022-09-27T16:14:28.5670252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:28.5670916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:28.5672294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:28.5672783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:29.0568043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:29.0590307Z 
2022-09-27T16:14:29.0590610Z Running tests...
2022-09-27T16:14:29.0591056Z ----------------------------------------------------------------------
2022-09-27T16:14:29.7626630Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:29.7627249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:29.7628678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:29.7629159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:30.2646960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:30.2669866Z 
2022-09-27T16:14:30.2670267Z Running tests...
2022-09-27T16:14:30.2688490Z ----------------------------------------------------------------------
2022-09-27T16:14:30.2689311Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:14:30.2689918Z 
2022-09-27T16:14:30.2690293Z ----------------------------------------------------------------------
2022-09-27T16:14:30.2690632Z Ran 1 test in 0.002s
2022-09-27T16:14:30.2690781Z 
2022-09-27T16:14:30.2690892Z OK (skipped=1)
2022-09-27T16:14:30.2691049Z 
2022-09-27T16:14:30.2691190Z Generating XML reports...
2022-09-27T16:14:30.2724986Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161430.xml
2022-09-27T16:14:30.6015598Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:30.6211047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59135
2022-09-27T16:14:30.6217087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59136
2022-09-27T16:14:32.2607038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:32.2607534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:32.2608851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:32.2609342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:32.2819367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:32.2819850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:32.2823372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:32.2823850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:32.3084421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:32.3084965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:32.3088039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:32.3088523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:32.7701938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:32.7723782Z 
2022-09-27T16:14:32.7723947Z Running tests...
2022-09-27T16:14:32.7724388Z ----------------------------------------------------------------------
2022-09-27T16:14:32.8253768Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:32.8254409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:32.8255180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:32.8255884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:32.8261708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:32.8263152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:34.4712413Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:34.4907920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59246
2022-09-27T16:14:34.4913618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59247
2022-09-27T16:14:35.8332452Z ok (6.774s)
2022-09-27T16:14:35.8332804Z 
2022-09-27T16:14:35.8333217Z ----------------------------------------------------------------------
2022-09-27T16:14:35.8333549Z Ran 1 test in 6.774s
2022-09-27T16:14:35.8333720Z 
2022-09-27T16:14:35.8333816Z OK
2022-09-27T16:14:35.8333950Z 
2022-09-27T16:14:35.8334085Z Generating XML reports...
2022-09-27T16:14:35.8368946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161429.xml
2022-09-27T16:14:36.1150879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:36.1151409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:36.1152625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:36.1153127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:36.1332416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:36.1332921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:36.1336282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:36.1336783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:36.6492320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:36.6528268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:36.6529099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:36.6592983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:36.6599153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:36.7543839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:37.8312561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:37.8313075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:37.8313982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:37.8314753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:38.3664351Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:38.3688578Z 
2022-09-27T16:14:38.3688745Z Running tests...
2022-09-27T16:14:38.3689622Z ----------------------------------------------------------------------
2022-09-27T16:14:38.6015540Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... ok (5.829s)
2022-09-27T16:14:38.6015895Z 
2022-09-27T16:14:38.6016262Z ----------------------------------------------------------------------
2022-09-27T16:14:38.6016619Z Ran 1 test in 5.829s
2022-09-27T16:14:38.6016786Z 
2022-09-27T16:14:38.6016881Z OK
2022-09-27T16:14:38.6017021Z 
2022-09-27T16:14:38.6017149Z Generating XML reports...
2022-09-27T16:14:38.6052502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161432.xml
2022-09-27T16:14:40.1024925Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:40.1221178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59398
2022-09-27T16:14:40.1227572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59399
2022-09-27T16:14:40.6425383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:40.6425903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:40.6427348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:40.6427835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:41.1767245Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:41.1789149Z 
2022-09-27T16:14:41.1789466Z Running tests...
2022-09-27T16:14:41.1789914Z ----------------------------------------------------------------------
2022-09-27T16:14:41.7757033Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:41.7757855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:41.7759303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:41.7759783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:41.7818872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:41.7819356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:41.7823513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:41.7823988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:42.3189061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:42.3189693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:42.3190766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:42.3191549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:42.3296751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:42.4204330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:42.7614623Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:42.7817469Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59477
2022-09-27T16:14:42.7822868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59478
2022-09-27T16:14:44.3854086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:44.3855081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:44.3856252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:44.3857213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:44.4329108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:44.4330080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:44.4331287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:44.4332243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:44.9259840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:44.9486172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:44.9487299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:44.9565593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:44.9573482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:45.0501389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:45.3344338Z ok (6.965s)
2022-09-27T16:14:45.3344565Z 
2022-09-27T16:14:45.3344966Z ----------------------------------------------------------------------
2022-09-27T16:14:45.3345313Z Ran 1 test in 6.965s
2022-09-27T16:14:45.3345488Z 
2022-09-27T16:14:45.3345590Z OK
2022-09-27T16:14:45.3345727Z 
2022-09-27T16:14:45.3345868Z Generating XML reports...
2022-09-27T16:14:45.3381134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161438.xml
2022-09-27T16:14:46.6755512Z STAGE:2022-09-27 16:14:46 59478:59478 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:14:46.7141578Z STAGE:2022-09-27 16:14:46 59477:59477 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:14:46.8709189Z STAGE:2022-09-27 16:14:46 59478:59478 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:46.8710441Z STAGE:2022-09-27 16:14:46 59477:59477 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:47.5133367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:47.5133895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:47.5135966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:47.5136448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:47.5931042Z ok (6.414s)
2022-09-27T16:14:47.5931375Z 
2022-09-27T16:14:47.5931929Z ----------------------------------------------------------------------
2022-09-27T16:14:47.5932446Z Ran 1 test in 6.414s
2022-09-27T16:14:47.5932620Z 
2022-09-27T16:14:47.5932697Z OK
2022-09-27T16:14:47.5932940Z 
2022-09-27T16:14:47.5933176Z Generating XML reports...
2022-09-27T16:14:47.5968595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161441.xml
2022-09-27T16:14:48.0358783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:48.0382416Z 
2022-09-27T16:14:48.0382719Z Running tests...
2022-09-27T16:14:48.0383424Z ----------------------------------------------------------------------
2022-09-27T16:14:49.6338523Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:49.6428376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:49.6428868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:49.6432209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:49.6432723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:49.6535855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59635
2022-09-27T16:14:49.6542511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59636
2022-09-27T16:14:50.1298228Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:50.1319833Z 
2022-09-27T16:14:50.1320332Z Running tests...
2022-09-27T16:14:50.1321057Z ----------------------------------------------------------------------
2022-09-27T16:14:51.2624062Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:51.2624676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:51.2625481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:51.2625950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:51.2872146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:51.2872640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:51.2875598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:51.2876098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:51.6465692Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:51.6658030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59703
2022-09-27T16:14:51.6664212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59704
2022-09-27T16:14:51.8252873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:51.8253429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:51.8254211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:51.8254936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:51.8361868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:51.9267735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:53.1990730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j0xpcep
2022-09-27T16:14:53.1991868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j0xpcep/_remote_module_non_scriptable.py
2022-09-27T16:14:53.2729032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g4in9f_
2022-09-27T16:14:53.2730413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g4in9f_/_remote_module_non_scriptable.py
2022-09-27T16:14:53.2950760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:53.2952065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:53.2953944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:53.2954894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:53.3119653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:53.3120961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:53.3122794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:53.3123714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:53.7471099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:53.7472110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:53.7494755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:53.7495706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:14:53.8359588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:14:53.8381249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:14:53.8382057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:53.8461416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:14:53.8467157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:14:53.9397345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:14:54.3655287Z ok (6.327s)
2022-09-27T16:14:54.3655520Z 
2022-09-27T16:14:54.3655913Z ----------------------------------------------------------------------
2022-09-27T16:14:54.3656260Z Ran 1 test in 6.327s
2022-09-27T16:14:54.3656441Z 
2022-09-27T16:14:54.3656518Z OK
2022-09-27T16:14:54.3656659Z 
2022-09-27T16:14:54.3656796Z Generating XML reports...
2022-09-27T16:14:54.3692181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161448.xml
2022-09-27T16:14:55.4987309Z STAGE:2022-09-27 16:14:55 59703:59703 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:14:55.5672686Z STAGE:2022-09-27 16:14:55 59704:59704 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:14:55.6746120Z STAGE:2022-09-27 16:14:55 59704:59704 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:55.6746919Z STAGE:2022-09-27 16:14:55 59703:59703 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:14:56.3770407Z ok (6.245s)
2022-09-27T16:14:56.3770814Z 
2022-09-27T16:14:56.3771486Z ----------------------------------------------------------------------
2022-09-27T16:14:56.3772131Z Ran 1 test in 6.245s
2022-09-27T16:14:56.3772441Z 
2022-09-27T16:14:56.3772617Z OK
2022-09-27T16:14:56.3772878Z 
2022-09-27T16:14:56.3773131Z Generating XML reports...
2022-09-27T16:14:56.3807421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161450.xml
2022-09-27T16:14:56.5426410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:56.5427379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:56.5429051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:56.5430007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:57.0603313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:14:57.0627666Z 
2022-09-27T16:14:57.0628375Z Running tests...
2022-09-27T16:14:57.0628895Z ----------------------------------------------------------------------
2022-09-27T16:14:58.4059022Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:14:58.4059652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:14:58.4060729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:14:58.4061266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:14:58.6848663Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:14:58.7045206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59874
2022-09-27T16:14:58.7051732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59875
2022-09-27T16:14:58.9103977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:14:58.9125929Z 
2022-09-27T16:14:58.9126268Z Running tests...
2022-09-27T16:14:58.9126769Z ----------------------------------------------------------------------
2022-09-27T16:14:58.9144151Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:14:58.9144834Z 
2022-09-27T16:14:58.9145445Z ----------------------------------------------------------------------
2022-09-27T16:14:58.9146120Z Ran 1 test in 0.002s
2022-09-27T16:14:58.9146442Z 
2022-09-27T16:14:58.9146651Z OK (skipped=1)
2022-09-27T16:14:58.9146953Z 
2022-09-27T16:14:58.9147167Z Generating XML reports...
2022-09-27T16:14:58.9180890Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161458.xml
2022-09-27T16:15:00.3429988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:00.3430544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:00.3431802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:00.3432295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:00.3941695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:00.3942185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:00.3945129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:00.3945633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:00.8001537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:00.8002039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:00.8004617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:00.8005117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:00.9200005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:00.9200910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:00.9201692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:00.9202603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:00.9308002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:01.0213939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:01.2910081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:01.2932707Z 
2022-09-27T16:15:01.2933104Z Running tests...
2022-09-27T16:15:01.2933581Z ----------------------------------------------------------------------
2022-09-27T16:15:01.2951067Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:15:01.2951404Z 
2022-09-27T16:15:01.2951687Z ----------------------------------------------------------------------
2022-09-27T16:15:01.2952013Z Ran 1 test in 0.002s
2022-09-27T16:15:01.2952182Z 
2022-09-27T16:15:01.2952296Z OK (skipped=1)
2022-09-27T16:15:01.2952453Z 
2022-09-27T16:15:01.2952573Z Generating XML reports...
2022-09-27T16:15:01.2986847Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161501.xml
2022-09-27T16:15:02.2564724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu717bzm6
2022-09-27T16:15:02.2566259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu717bzm6/_remote_module_non_scriptable.py
2022-09-27T16:15:02.3677454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m2j34ct
2022-09-27T16:15:02.3679148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m2j34ct/_remote_module_non_scriptable.py
2022-09-27T16:15:02.8500301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:02.8500836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:02.8522882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:02.8523387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:03.2934034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:03.2934545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:03.2936912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:03.2937409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:03.5154797Z ok (6.452s)
2022-09-27T16:15:03.5154991Z 
2022-09-27T16:15:03.5155394Z ----------------------------------------------------------------------
2022-09-27T16:15:03.5155754Z Ran 1 test in 6.453s
2022-09-27T16:15:03.5155930Z 
2022-09-27T16:15:03.5156030Z OK
2022-09-27T16:15:03.5156166Z 
2022-09-27T16:15:03.5156287Z Generating XML reports...
2022-09-27T16:15:03.5192314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161457.xml
2022-09-27T16:15:03.7940487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:03.7963179Z 
2022-09-27T16:15:03.7963575Z Running tests...
2022-09-27T16:15:03.7964061Z ----------------------------------------------------------------------
2022-09-27T16:15:03.7981507Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:15:03.7982115Z 
2022-09-27T16:15:03.7982416Z ----------------------------------------------------------------------
2022-09-27T16:15:03.7982858Z Ran 1 test in 0.002s
2022-09-27T16:15:03.7983108Z 
2022-09-27T16:15:03.7983225Z OK (skipped=1)
2022-09-27T16:15:03.7983383Z 
2022-09-27T16:15:03.7983514Z Generating XML reports...
2022-09-27T16:15:03.8017228Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161503.xml
2022-09-27T16:15:05.5560000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:05.5560706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:05.5561802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:05.5562276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:05.8200675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:05.8201443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:05.8204090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:05.8204596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:06.0409800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:15:06.0431877Z 
2022-09-27T16:15:06.0432313Z Running tests...
2022-09-27T16:15:06.0432746Z ----------------------------------------------------------------------
2022-09-27T16:15:06.3167463Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:06.3190835Z 
2022-09-27T16:15:06.3191087Z Running tests...
2022-09-27T16:15:06.3191518Z ----------------------------------------------------------------------
2022-09-27T16:15:06.3214934Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:15:06.3215691Z 
2022-09-27T16:15:06.3216102Z ----------------------------------------------------------------------
2022-09-27T16:15:06.3216460Z Ran 1 test in 0.002s
2022-09-27T16:15:06.3216633Z 
2022-09-27T16:15:06.3216749Z OK (skipped=1)
2022-09-27T16:15:06.3216893Z 
2022-09-27T16:15:06.3217017Z Generating XML reports...
2022-09-27T16:15:06.3258123Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161506.xml
2022-09-27T16:15:07.6199829Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:07.6392194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60120
2022-09-27T16:15:07.6398197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60121
2022-09-27T16:15:08.2477321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:08.2477841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:08.2480069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:08.2481023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:08.7988430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:08.8011704Z 
2022-09-27T16:15:08.8012017Z Running tests...
2022-09-27T16:15:08.8012469Z ----------------------------------------------------------------------
2022-09-27T16:15:08.8029094Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-09-27T16:15:08.8029880Z 
2022-09-27T16:15:08.8030192Z ----------------------------------------------------------------------
2022-09-27T16:15:08.8030530Z Ran 1 test in 0.002s
2022-09-27T16:15:08.8030694Z 
2022-09-27T16:15:08.8030804Z OK (skipped=1)
2022-09-27T16:15:08.8030964Z 
2022-09-27T16:15:08.8031095Z Generating XML reports...
2022-09-27T16:15:08.8065853Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161508.xml
2022-09-27T16:15:09.2423856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:09.2424389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:09.2426110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:09.2426644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:09.2582319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:09.2582792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:09.2586462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:09.2586974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:09.7778792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:09.7780043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:09.7782168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:09.7783790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:09.7863561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:09.7866643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:10.1458886Z skip: Need at least 4 CUDA devices (4.102s)
2022-09-27T16:15:10.1459348Z 
2022-09-27T16:15:10.1459997Z ----------------------------------------------------------------------
2022-09-27T16:15:10.1460653Z Ran 1 test in 4.102s
2022-09-27T16:15:10.1460975Z 
2022-09-27T16:15:10.1461176Z OK (skipped=1)
2022-09-27T16:15:10.1461472Z 
2022-09-27T16:15:10.1461698Z Generating XML reports...
2022-09-27T16:15:10.1496511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161506.xml
2022-09-27T16:15:10.6988558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:10.6989534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:10.6990679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:10.6991603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:11.2306743Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:11.2352368Z 
2022-09-27T16:15:11.2352969Z Running tests...
2022-09-27T16:15:11.2354009Z ----------------------------------------------------------------------
2022-09-27T16:15:11.2388165Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.003s)
2022-09-27T16:15:11.2388978Z 
2022-09-27T16:15:11.2389596Z ----------------------------------------------------------------------
2022-09-27T16:15:11.2390343Z Ran 1 test in 0.004s
2022-09-27T16:15:11.2391055Z 
2022-09-27T16:15:11.2391315Z OK (skipped=1)
2022-09-27T16:15:11.2391983Z 
2022-09-27T16:15:11.2392382Z Generating XML reports...
2022-09-27T16:15:11.2456234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161511.xml
2022-09-27T16:15:12.1189826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:12.1190338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:12.1195020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:12.1195532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:12.5966266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:15:12.5989429Z 
2022-09-27T16:15:12.5989680Z Running tests...
2022-09-27T16:15:12.5990119Z ----------------------------------------------------------------------
2022-09-27T16:15:13.1057896Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:13.1058573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:13.1059885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:13.1060406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:13.6051420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:13.6074708Z 
2022-09-27T16:15:13.6075117Z Running tests...
2022-09-27T16:15:13.6075602Z ----------------------------------------------------------------------
2022-09-27T16:15:14.1356026Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:14.1555169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60297
2022-09-27T16:15:14.1558917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60298
2022-09-27T16:15:15.2510336Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:15.2702386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60362
2022-09-27T16:15:15.2708496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60363
2022-09-27T16:15:15.7710117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:15.7710879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:15.7712051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:15.7712522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:15.8315822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:15.8316325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:15.8319028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:15.8319523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:16.4089016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:16.4089553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:16.4090346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:16.4091050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:16.4197695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:16.5102399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:16.8624087Z skip: Need at least 4 CUDA devices (4.263s)
2022-09-27T16:15:16.8624378Z 
2022-09-27T16:15:16.8624768Z ----------------------------------------------------------------------
2022-09-27T16:15:16.8625111Z Ran 1 test in 4.263s
2022-09-27T16:15:16.8625518Z 
2022-09-27T16:15:16.8625648Z OK (skipped=1)
2022-09-27T16:15:16.8625794Z 
2022-09-27T16:15:16.8626938Z Generating XML reports...
2022-09-27T16:15:16.8661880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161512.xml
2022-09-27T16:15:16.9158430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:16.9158933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:16.9160554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:16.9161314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:16.9176054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:16.9176715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:16.9180854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:16.9181602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:17.4338808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:17.4339720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:17.4340885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:17.4439960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:17.4446934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:17.5356004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:18.9939791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:18.9940330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:18.9943342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:18.9943993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:19.5247707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:15:19.5271343Z 
2022-09-27T16:15:19.5271739Z Running tests...
2022-09-27T16:15:19.5272238Z ----------------------------------------------------------------------
2022-09-27T16:15:20.9830578Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... ok (7.375s)
2022-09-27T16:15:20.9830906Z 
2022-09-27T16:15:20.9831300Z ----------------------------------------------------------------------
2022-09-27T16:15:20.9831621Z Ran 1 test in 7.375s
2022-09-27T16:15:20.9831793Z 
2022-09-27T16:15:20.9831889Z OK
2022-09-27T16:15:20.9832028Z 
2022-09-27T16:15:20.9832166Z Generating XML reports...
2022-09-27T16:15:20.9867150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161513.xml
2022-09-27T16:15:21.2348292Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:21.2544081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60493
2022-09-27T16:15:21.2549745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60494
2022-09-27T16:15:22.9478941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:22.9479461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:22.9482138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:22.9482648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:22.9489007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:22.9489486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:22.9493053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:22.9493523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:23.0655119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:23.0655617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:23.0656671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:23.0657135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:23.4801011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:23.4802022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:23.4802813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:23.4803503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:23.4809734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:23.4810227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:23.5474878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:23.5497319Z 
2022-09-27T16:15:23.5497503Z Running tests...
2022-09-27T16:15:23.5497921Z ----------------------------------------------------------------------
2022-09-27T16:15:23.5518569Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-09-27T16:15:23.5518897Z 
2022-09-27T16:15:23.5519173Z ----------------------------------------------------------------------
2022-09-27T16:15:23.5519509Z Ran 1 test in 0.002s
2022-09-27T16:15:23.5519658Z 
2022-09-27T16:15:23.5519767Z OK (skipped=1)
2022-09-27T16:15:23.5519923Z 
2022-09-27T16:15:23.5520051Z Generating XML reports...
2022-09-27T16:15:23.5554192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161523.xml
2022-09-27T16:15:24.8740304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8ne3ksh
2022-09-27T16:15:24.8741206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8ne3ksh/_remote_module_non_scriptable.py
2022-09-27T16:15:24.8744171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphdtrlhm4
2022-09-27T16:15:24.8746864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphdtrlhm4/_remote_module_non_scriptable.py
2022-09-27T16:15:25.3451479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:25.3452022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:15:25.5619833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:25.5620356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:25.5622732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:25.5623505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:25.8651569Z ok (6.338s)
2022-09-27T16:15:25.8651782Z 
2022-09-27T16:15:25.8652171Z ----------------------------------------------------------------------
2022-09-27T16:15:25.8652515Z Ran 1 test in 6.338s
2022-09-27T16:15:25.8652681Z 
2022-09-27T16:15:25.8652779Z OK
2022-09-27T16:15:25.8652915Z 
2022-09-27T16:15:25.8653048Z Generating XML reports...
2022-09-27T16:15:25.8688688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161519.xml
2022-09-27T16:15:26.0769552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:26.0791729Z 
2022-09-27T16:15:26.0791991Z Running tests...
2022-09-27T16:15:26.0792428Z ----------------------------------------------------------------------
2022-09-27T16:15:27.6553371Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:27.6745454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60673
2022-09-27T16:15:27.6751513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60674
2022-09-27T16:15:27.8538780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:27.8540088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:27.8541224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:27.8542120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:28.4412390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:15:28.4450339Z 
2022-09-27T16:15:28.4450832Z Running tests...
2022-09-27T16:15:28.4451330Z ----------------------------------------------------------------------
2022-09-27T16:15:29.2758405Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:29.2759020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:29.2760119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:29.2760780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:29.3526449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:29.3526954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:29.3528088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:29.3528574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:29.7836354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:29.8720476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:29.8721319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:29.8747766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:29.8754654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:29.9734722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:29.9938174Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:30.0137587Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60750
2022-09-27T16:15:30.0144252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60751
2022-09-27T16:15:31.2214768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_r_8hn4c
2022-09-27T16:15:31.2215645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_r_8hn4c/_remote_module_non_scriptable.py
2022-09-27T16:15:31.3507115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptuxodfdl
2022-09-27T16:15:31.3508232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptuxodfdl/_remote_module_non_scriptable.py
2022-09-27T16:15:31.6343494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:31.6344452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:31.6345639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:31.6346589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:31.6698171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:31.6699138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:31.6701091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:31.6702008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:32.1993501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:32.1994086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:32.1994872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:32.1995575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:32.2102499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:32.3006542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:32.3213925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:15:32.3214539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:15:32.3215351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:32.3216205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:32.3311586Z STAGE:2022-09-27 16:15:32 60751:60751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:32.3312219Z STAGE:2022-09-27 16:15:32 60750:60750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:32.3325888Z STAGE:2022-09-27 16:15:32 60750:60750 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:15:32 60751:60751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:32.3326306Z 
2022-09-27T16:15:32.3342165Z STAGE:2022-09-27 16:15:32 60751:60751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:32.3349370Z STAGE:2022-09-27 16:15:32 60750:60750 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:32.3362484Z STAGE:2022-09-27 16:15:32 60750:60750 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:32.3363098Z STAGE:2022-09-27 16:15:32 60751:60751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:32.7213201Z ok (4.276s)
2022-09-27T16:15:32.7213439Z 
2022-09-27T16:15:32.7213828Z ----------------------------------------------------------------------
2022-09-27T16:15:32.7214146Z Ran 1 test in 4.276s
2022-09-27T16:15:32.7214605Z 
2022-09-27T16:15:32.7214720Z OK
2022-09-27T16:15:32.7214865Z 
2022-09-27T16:15:32.7215006Z Generating XML reports...
2022-09-27T16:15:32.7251253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161528.xml
2022-09-27T16:15:33.0868539Z ok (7.007s)
2022-09-27T16:15:33.0868778Z 
2022-09-27T16:15:33.0869186Z ----------------------------------------------------------------------
2022-09-27T16:15:33.0869529Z Ran 1 test in 7.008s
2022-09-27T16:15:33.0869698Z 
2022-09-27T16:15:33.0869777Z OK
2022-09-27T16:15:33.0869915Z 
2022-09-27T16:15:33.0870048Z Generating XML reports...
2022-09-27T16:15:33.0906822Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161526.xml
2022-09-27T16:15:34.9031261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:34.9031808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:34.9034118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:34.9034924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:35.2325497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:35.2325995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:35.2328367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:35.2328852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:35.3987847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:15:35.4011660Z 
2022-09-27T16:15:35.4012155Z Running tests...
2022-09-27T16:15:35.4012653Z ----------------------------------------------------------------------
2022-09-27T16:15:35.7299969Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:35.7323191Z 
2022-09-27T16:15:35.7323521Z Running tests...
2022-09-27T16:15:35.7323952Z ----------------------------------------------------------------------
2022-09-27T16:15:37.0003115Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:37.0201387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60911
2022-09-27T16:15:37.0208649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60912
2022-09-27T16:15:37.3117389Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:37.3314421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60914
2022-09-27T16:15:37.3320923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60915
2022-09-27T16:15:38.7384242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:38.7384771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:38.7389649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:38.7390133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:38.7545691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:38.7546161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:38.7549770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:38.7550516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:38.9874420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:38.9875400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:38.9876635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:38.9877521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:39.0128139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:39.0129109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:39.0131066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:39.0132028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:39.2814723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:39.2815304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:39.2816343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:39.2817032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:39.2923199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:39.3824539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:39.4031570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:15:39.4032361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:15:39.4033110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:39.4033873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:39.4127661Z STAGE:2022-09-27 16:15:39 60912:60912 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:39.4128260Z STAGE:2022-09-27 16:15:39 60911:60911 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:39.4142009Z STAGE:2022-09-27 16:15:39 60912:60912 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:15:39 60911:60911 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:39.4142431Z 
2022-09-27T16:15:39.4158996Z STAGE:2022-09-27 16:15:39 60912:60912 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:39.4165797Z STAGE:2022-09-27 16:15:39 60911:60911 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:39.4179388Z STAGE:2022-09-27 16:15:39 60911:60911 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:39.4180003Z STAGE:2022-09-27 16:15:39 60912:60912 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:39.4798749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:39.5213067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:39.5213892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:39.5307328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:39.5313843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:39.6227808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:39.8289714Z ok (4.427s)
2022-09-27T16:15:39.8290227Z 
2022-09-27T16:15:39.8290797Z ----------------------------------------------------------------------
2022-09-27T16:15:39.8291198Z Ran 1 test in 4.428s
2022-09-27T16:15:39.8291369Z 
2022-09-27T16:15:39.8291477Z OK
2022-09-27T16:15:39.8291610Z 
2022-09-27T16:15:39.8291747Z Generating XML reports...
2022-09-27T16:15:39.8339752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161535.xml
2022-09-27T16:15:40.9098007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ye6h831
2022-09-27T16:15:40.9098616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ye6h831/_remote_module_non_scriptable.py
2022-09-27T16:15:40.9685883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08np2s3l
2022-09-27T16:15:40.9687038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08np2s3l/_remote_module_non_scriptable.py
2022-09-27T16:15:41.9816670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:41.9817561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:41.9819299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:41.9819798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:42.5102052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:15:42.5124875Z 
2022-09-27T16:15:42.5125103Z Running tests...
2022-09-27T16:15:42.5125545Z ----------------------------------------------------------------------
2022-09-27T16:15:42.5383433Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:15:42.5392858Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:15:43.1442300Z ok (7.412s)
2022-09-27T16:15:43.1442522Z 
2022-09-27T16:15:43.1442937Z ----------------------------------------------------------------------
2022-09-27T16:15:43.1443309Z Ran 1 test in 7.412s
2022-09-27T16:15:43.1443477Z 
2022-09-27T16:15:43.1443553Z OK
2022-09-27T16:15:43.1443690Z 
2022-09-27T16:15:43.1443830Z Generating XML reports...
2022-09-27T16:15:43.1479113Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161535.xml
2022-09-27T16:15:44.2020025Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:44.2212958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61118
2022-09-27T16:15:44.2219113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61119
2022-09-27T16:15:45.3594709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:45.3595669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:45.3600917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:45.3601788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:45.8176790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:45.8177293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:45.8178628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:45.8179127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:45.8435499Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:45.8458778Z 
2022-09-27T16:15:45.8458997Z Running tests...
2022-09-27T16:15:45.8459430Z ----------------------------------------------------------------------
2022-09-27T16:15:45.9011662Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:45.9012573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:45.9014859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:45.9015332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:46.4575154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:46.4575704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:46.4576515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:46.4577223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:46.4683468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:46.5590519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:46.5797788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:15:46.5798317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:15:46.5799084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:46.5799794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:46.5887586Z STAGE:2022-09-27 16:15:46 61119:61119 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:46.5891409Z STAGE:2022-09-27 16:15:46 61118:61118 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:46.5905265Z STAGE:2022-09-27 16:15:46 61119:61119 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:15:46 61118:61118 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:46.5905740Z 
2022-09-27T16:15:46.5921372Z STAGE:2022-09-27 16:15:46 61119:61119 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:46.5929579Z STAGE:2022-09-27 16:15:46 61118:61118 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:46.5943099Z STAGE:2022-09-27 16:15:46 61118:61118 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:46.5943698Z STAGE:2022-09-27 16:15:46 61119:61119 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:47.0288206Z ok (4.516s)
2022-09-27T16:15:47.0288461Z 
2022-09-27T16:15:47.0288846Z ----------------------------------------------------------------------
2022-09-27T16:15:47.0289189Z Ran 1 test in 4.516s
2022-09-27T16:15:47.0289352Z 
2022-09-27T16:15:47.0289448Z OK
2022-09-27T16:15:47.0289585Z 
2022-09-27T16:15:47.0289720Z Generating XML reports...
2022-09-27T16:15:47.0325591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161542.xml
2022-09-27T16:15:47.3917830Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:47.4113545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61236
2022-09-27T16:15:47.4119581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61237
2022-09-27T16:15:49.0535313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:49.0535832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:49.0536907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:49.0537730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:49.0538677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:49.0539163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:49.0539758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:49.0540221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:49.1295754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:49.1296281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:49.1298468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:49.1298966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:49.5814715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:49.5821765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:49.5822833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:49.5915803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:49.5922489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:49.6182439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:15:49.6206571Z 
2022-09-27T16:15:49.6206981Z Running tests...
2022-09-27T16:15:49.6834929Z ----------------------------------------------------------------------
2022-09-27T16:15:49.6835492Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:49.6944594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:15:49.6945195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:15:49.6945927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:49.6946610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:49.6948747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:15:49.7050908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:15:49.7051664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:15:49.7052352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:15:51.0677632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkp405lgh
2022-09-27T16:15:51.0678268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkp405lgh/_remote_module_non_scriptable.py
2022-09-27T16:15:51.0679550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpob42rx0l
2022-09-27T16:15:51.0683004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpob42rx0l/_remote_module_non_scriptable.py
2022-09-27T16:15:51.2843929Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:51.3040197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61360
2022-09-27T16:15:51.3046833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61361
2022-09-27T16:15:52.8938084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:52.8938601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:52.8939991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:52.8940494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:52.9154686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:52.9155161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:52.9159016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:52.9159515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:53.4393615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:15:53.4394194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:15:53.4395000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:53.4395714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:15:53.4501888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:15:53.5404842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:15:53.5612714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:15:53.5613248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:15:53.5613939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:53.5614645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:15:53.5688540Z STAGE:2022-09-27 16:15:53 61360:61360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:53.5775277Z STAGE:2022-09-27 16:15:53 61361:61361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:53.5788193Z STAGE:2022-09-27 16:15:53 61361:61361 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:53.5788798Z STAGE:2022-09-27 16:15:53 61360:61360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:53.5805031Z STAGE:2022-09-27 16:15:53 61361:61361 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:53.5812080Z STAGE:2022-09-27 16:15:53 61360:61360 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:15:53.5825925Z STAGE:2022-09-27 16:15:53 61361:61361 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:15:53 61360:61360 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:15:53.5826346Z 
2022-09-27T16:15:54.0111566Z ok (4.390s)
2022-09-27T16:15:54.0111764Z 
2022-09-27T16:15:54.0112157Z ----------------------------------------------------------------------
2022-09-27T16:15:54.0112497Z Ran 1 test in 4.390s
2022-09-27T16:15:54.0112666Z 
2022-09-27T16:15:54.0112764Z OK
2022-09-27T16:15:54.0112900Z 
2022-09-27T16:15:54.0113015Z Generating XML reports...
2022-09-27T16:15:54.0149259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161549.xml
2022-09-27T16:15:56.0215892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:56.0216417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:56.0218086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:56.0218559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:56.5360021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:15:56.5384399Z 
2022-09-27T16:15:56.5384795Z Running tests...
2022-09-27T16:15:56.5385282Z ----------------------------------------------------------------------
2022-09-27T16:15:57.0303736Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... ok (11.184s)
2022-09-27T16:15:57.0304058Z 
2022-09-27T16:15:57.0304447Z ----------------------------------------------------------------------
2022-09-27T16:15:57.0304792Z Ran 1 test in 11.184s
2022-09-27T16:15:57.0304962Z 
2022-09-27T16:15:57.0305058Z OK
2022-09-27T16:15:57.0305194Z 
2022-09-27T16:15:57.0305321Z Generating XML reports...
2022-09-27T16:15:57.0340916Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161545.xml
2022-09-27T16:15:58.1834968Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:15:58.2038707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61511
2022-09-27T16:15:58.2166103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61512
2022-09-27T16:15:59.1046363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:59.1046887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:59.1049031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:59.1049534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:59.5861548Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:15:59.5883296Z 
2022-09-27T16:15:59.5883548Z Running tests...
2022-09-27T16:15:59.5883982Z ----------------------------------------------------------------------
2022-09-27T16:15:59.8713557Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:59.8714176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:59.8715238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:59.8715736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:15:59.8877753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:15:59.8878253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:15:59.8880599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:15:59.8881324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:00.4184324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:00.4184887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:00.4185666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:00.4186420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:00.4292863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:00.5198662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:00.9231706Z skip: Skipped due to small world size. (4.384s)
2022-09-27T16:16:00.9231960Z 
2022-09-27T16:16:00.9232358Z ----------------------------------------------------------------------
2022-09-27T16:16:00.9232686Z Ran 1 test in 4.385s
2022-09-27T16:16:00.9232852Z 
2022-09-27T16:16:00.9232970Z OK (skipped=1)
2022-09-27T16:16:00.9233128Z 
2022-09-27T16:16:00.9233259Z Generating XML reports...
2022-09-27T16:16:00.9268307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161556.xml
2022-09-27T16:16:01.1569717Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:01.1769484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61588
2022-09-27T16:16:01.1774553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61589
2022-09-27T16:16:02.8957429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:02.8957967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:02.8958920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:02.8959414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:02.9056101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:02.9056561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:02.9059945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:02.9060442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:03.1303437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:03.1304151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:03.1306419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:03.1306915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:03.4125364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:03.4157718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:03.4158624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:03.4226886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:03.4233378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:03.5168966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:03.5350235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:16:03.5350737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:16:03.5351427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:16:03.5352118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:16:03.5353833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:16:03.5354330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:16:03.5355001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:16:03.5355839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:16:03.6177013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:03.6199576Z 
2022-09-27T16:16:03.6200031Z Running tests...
2022-09-27T16:16:03.6200748Z ----------------------------------------------------------------------
2022-09-27T16:16:04.8866162Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0bgxnfe
2022-09-27T16:16:04.8866862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0bgxnfe/_remote_module_non_scriptable.py
2022-09-27T16:16:04.9396904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpail96yfv
2022-09-27T16:16:04.9397921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpail96yfv/_remote_module_non_scriptable.py
2022-09-27T16:16:05.2033666Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:05.2226193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61712
2022-09-27T16:16:05.2232795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61713
2022-09-27T16:16:06.8539052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:06.8539563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:06.8541030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:06.8541500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:06.8638646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:06.8639122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:06.8642808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:06.8643281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:07.4010211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:07.4010769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:07.4011554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:07.4012492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:07.4018565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:07.4019401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:07.7291916Z skip: Skipped due to small world size. (4.109s)
2022-09-27T16:16:07.7292183Z 
2022-09-27T16:16:07.7292560Z ----------------------------------------------------------------------
2022-09-27T16:16:07.7292882Z Ran 1 test in 4.109s
2022-09-27T16:16:07.7293049Z 
2022-09-27T16:16:07.7293157Z OK (skipped=1)
2022-09-27T16:16:07.7293314Z 
2022-09-27T16:16:07.7293448Z Generating XML reports...
2022-09-27T16:16:07.7328876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161603.xml
2022-09-27T16:16:09.7163504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:09.7164043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:09.7166372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:09.7167148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:10.2130313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:10.2152390Z 
2022-09-27T16:16:10.2152644Z Running tests...
2022-09-27T16:16:10.2153064Z ----------------------------------------------------------------------
2022-09-27T16:16:10.8964743Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... ok (11.308s)
2022-09-27T16:16:10.8965064Z 
2022-09-27T16:16:10.8965437Z ----------------------------------------------------------------------
2022-09-27T16:16:10.8965777Z Ran 1 test in 11.308s
2022-09-27T16:16:10.8965931Z 
2022-09-27T16:16:10.8966032Z OK
2022-09-27T16:16:10.8966168Z 
2022-09-27T16:16:10.8966294Z Generating XML reports...
2022-09-27T16:16:10.9001951Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161559.xml
2022-09-27T16:16:11.9634102Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:11.9827283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61821
2022-09-27T16:16:11.9833300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61822
2022-09-27T16:16:13.6150922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:13.6151436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:13.6153711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:13.6154192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:13.6333032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:13.6333510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:13.6336903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:13.6337399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:14.1722293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:14.1722854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:14.1723640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:14.1724330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:14.1731395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:14.1731911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:14.4895458Z skip: Skipped due to small world size. (4.274s)
2022-09-27T16:16:14.4895722Z 
2022-09-27T16:16:14.4896069Z ----------------------------------------------------------------------
2022-09-27T16:16:14.4896412Z Ran 1 test in 4.274s
2022-09-27T16:16:14.4896581Z 
2022-09-27T16:16:14.4896696Z OK (skipped=1)
2022-09-27T16:16:14.4896850Z 
2022-09-27T16:16:14.4896987Z Generating XML reports...
2022-09-27T16:16:14.4933103Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161610.xml
2022-09-27T16:16:16.4891844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:16.4892515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:16.4894899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:16.4895396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:17.0471776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:17.0494148Z 
2022-09-27T16:16:17.0494554Z Running tests...
2022-09-27T16:16:17.0495024Z ----------------------------------------------------------------------
2022-09-27T16:16:18.5809831Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:18.6000373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61930
2022-09-27T16:16:18.6006975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61931
2022-09-27T16:16:20.1984154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:20.1984697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:20.1985579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:20.1986090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:20.2598036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:20.2598529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:20.2601899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:20.2602396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:20.7777912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:20.7778532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:20.7779318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:20.7780037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:20.7886228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:20.8792312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:21.2069260Z skip: Skipped due to small world size. (4.157s)
2022-09-27T16:16:21.2069559Z 
2022-09-27T16:16:21.2070014Z ----------------------------------------------------------------------
2022-09-27T16:16:21.2070358Z Ran 1 test in 4.157s
2022-09-27T16:16:21.2070531Z 
2022-09-27T16:16:21.2070644Z OK (skipped=1)
2022-09-27T16:16:21.2070786Z 
2022-09-27T16:16:21.2070913Z Generating XML reports...
2022-09-27T16:16:21.2181543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161617.xml
2022-09-27T16:16:23.2292534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:23.2293069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:23.2294178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:23.2294673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:23.7368075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:23.7390223Z 
2022-09-27T16:16:23.7390485Z Running tests...
2022-09-27T16:16:23.7390916Z ----------------------------------------------------------------------
2022-09-27T16:16:25.2616879Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:25.2807635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62039
2022-09-27T16:16:25.2814400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62040
2022-09-27T16:16:26.9415633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:26.9416612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:26.9417813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:26.9418777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:26.9598392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:26.9598890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:26.9602731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:26.9603728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:27.4871733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:27.4872826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:27.4873928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:27.4874646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:27.4978919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:27.5043722Z STAGE:2022-09-27 16:16:27 62040:62040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:27.5882893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:27.5947231Z STAGE:2022-09-27 16:16:27 62039:62039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:27.5961460Z STAGE:2022-09-27 16:16:27 62040:62040 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:27.5962675Z STAGE:2022-09-27 16:16:27 62039:62039 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:27.5978138Z STAGE:2022-09-27 16:16:27 62040:62040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:27.5987318Z STAGE:2022-09-27 16:16:27 62039:62039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:27.6002069Z STAGE:2022-09-27 16:16:27 62039:62039 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:16:27 62040:62040 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:27.6002899Z 
2022-09-27T16:16:27.9878179Z ok (4.248s)
2022-09-27T16:16:27.9878400Z 
2022-09-27T16:16:27.9878803Z ----------------------------------------------------------------------
2022-09-27T16:16:27.9879144Z Ran 1 test in 4.249s
2022-09-27T16:16:27.9879324Z 
2022-09-27T16:16:27.9879419Z OK
2022-09-27T16:16:27.9879554Z 
2022-09-27T16:16:27.9879672Z Generating XML reports...
2022-09-27T16:16:27.9916492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161623.xml
2022-09-27T16:16:29.9862996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:29.9863543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:29.9864941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:29.9865503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:30.4958588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:30.4982123Z 
2022-09-27T16:16:30.4982491Z Running tests...
2022-09-27T16:16:30.4983209Z ----------------------------------------------------------------------
2022-09-27T16:16:32.0348532Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:32.0544524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62152
2022-09-27T16:16:32.0551208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62153
2022-09-27T16:16:33.6362970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:33.6363478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:33.6365085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:33.6365772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:33.6553269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:33.6554016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:33.6557205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:33.6557970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:34.1826066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:34.1826637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:34.1827413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:34.1828143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:34.1833888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:34.1834775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:34.1928952Z STAGE:2022-09-27 16:16:34 62153:62153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:34.1929793Z STAGE:2022-09-27 16:16:34 62152:62152 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:34.1942954Z STAGE:2022-09-27 16:16:34 62152:62152 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:16:34 62153:62153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:34.1943508Z 
2022-09-27T16:16:34.1958798Z STAGE:2022-09-27 16:16:34 62153:62153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:34.1966742Z STAGE:2022-09-27 16:16:34 62152:62152 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:34.1979800Z STAGE:2022-09-27 16:16:34 62152:62152 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:16:34 62153:62153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:34.1980388Z 
2022-09-27T16:16:34.5612889Z ok (4.063s)
2022-09-27T16:16:34.5613115Z 
2022-09-27T16:16:34.5613510Z ----------------------------------------------------------------------
2022-09-27T16:16:34.5613837Z Ran 1 test in 4.063s
2022-09-27T16:16:34.5614007Z 
2022-09-27T16:16:34.5614101Z OK
2022-09-27T16:16:34.5614238Z 
2022-09-27T16:16:34.5614376Z Generating XML reports...
2022-09-27T16:16:34.5650277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161630.xml
2022-09-27T16:16:36.5780983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:36.5781544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:36.5782835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:36.5783714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:37.0939231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:37.0962600Z 
2022-09-27T16:16:37.0962991Z Running tests...
2022-09-27T16:16:37.0963431Z ----------------------------------------------------------------------
2022-09-27T16:16:37.0984722Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-09-27T16:16:37.0985343Z 
2022-09-27T16:16:37.0985942Z ----------------------------------------------------------------------
2022-09-27T16:16:37.0986463Z Ran 1 test in 0.002s
2022-09-27T16:16:37.0986630Z 
2022-09-27T16:16:37.0986757Z OK (skipped=1)
2022-09-27T16:16:37.0986913Z 
2022-09-27T16:16:37.0987038Z Generating XML reports...
2022-09-27T16:16:37.1021354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161637.xml
2022-09-27T16:16:38.9212976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:38.9213485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:38.9215003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:38.9215826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:39.4131553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:39.4152873Z 
2022-09-27T16:16:39.4153326Z Running tests...
2022-09-27T16:16:39.4154289Z ----------------------------------------------------------------------
2022-09-27T16:16:40.9377801Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:40.9566859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62298
2022-09-27T16:16:40.9573091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62299
2022-09-27T16:16:42.5666949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:42.5667681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:42.5669017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:42.5669489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:42.6027596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:42.6028355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:42.6031544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:42.6032037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:43.1246876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:43.1247449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:43.1248250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:43.1248942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:43.1256232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:43.1256764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:43.1353142Z STAGE:2022-09-27 16:16:43 62298:62298 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:43.1353960Z STAGE:2022-09-27 16:16:43 62299:62299 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:43.1367220Z STAGE:2022-09-27 16:16:43 62299:62299 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:16:43 62298:62298 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:43.1367631Z 
2022-09-27T16:16:43.1382682Z STAGE:2022-09-27 16:16:43 62299:62299 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:43.1390186Z STAGE:2022-09-27 16:16:43 62298:62298 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:43.1403666Z STAGE:2022-09-27 16:16:43 62298:62298 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:43.1404249Z STAGE:2022-09-27 16:16:43 62299:62299 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:43.4631015Z ok (4.047s)
2022-09-27T16:16:43.4631235Z 
2022-09-27T16:16:43.4631671Z ----------------------------------------------------------------------
2022-09-27T16:16:43.4632015Z Ran 1 test in 4.048s
2022-09-27T16:16:43.4632165Z 
2022-09-27T16:16:43.4632259Z OK
2022-09-27T16:16:43.4632612Z 
2022-09-27T16:16:43.4632755Z Generating XML reports...
2022-09-27T16:16:43.4670341Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161639.xml
2022-09-27T16:16:45.4342917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:45.4343439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:45.4344381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:45.4344895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:45.9493641Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:45.9517370Z 
2022-09-27T16:16:45.9517581Z Running tests...
2022-09-27T16:16:45.9518015Z ----------------------------------------------------------------------
2022-09-27T16:16:45.9547780Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-09-27T16:16:45.9548106Z 
2022-09-27T16:16:45.9548367Z ----------------------------------------------------------------------
2022-09-27T16:16:45.9548699Z Ran 1 test in 0.003s
2022-09-27T16:16:45.9548864Z 
2022-09-27T16:16:45.9548981Z OK (skipped=1)
2022-09-27T16:16:45.9549137Z 
2022-09-27T16:16:45.9549262Z Generating XML reports...
2022-09-27T16:16:45.9583921Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161645.xml
2022-09-27T16:16:47.8237071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:47.8237589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:47.8238639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:47.8239126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:48.3283424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:48.3307111Z 
2022-09-27T16:16:48.3307268Z Running tests...
2022-09-27T16:16:48.3308149Z ----------------------------------------------------------------------
2022-09-27T16:16:49.8890737Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:16:49.9078154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62444
2022-09-27T16:16:49.9084568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62445
2022-09-27T16:16:51.4532281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:51.4533652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:51.4534856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:51.4535810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:51.5314717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:51.5315583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:51.5316933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:51.5317793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:52.0623336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:16:52.0624397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:16:52.0625832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:52.0626739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:16:52.0633040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:16:52.0634282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:16:52.0713021Z STAGE:2022-09-27 16:16:52 62445:62445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:52.0719824Z STAGE:2022-09-27 16:16:52 62444:62444 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:52.0734627Z STAGE:2022-09-27 16:16:52 62445:62445 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:16:52 62444:62444 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:52.0735396Z 
2022-09-27T16:16:52.0750856Z STAGE:2022-09-27 16:16:52 62445:62445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:52.0756713Z STAGE:2022-09-27 16:16:52 62444:62444 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:16:52.0771889Z STAGE:2022-09-27 16:16:52 62444:62444 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:16:52 62445:62445 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:16:52.0772695Z 
2022-09-27T16:16:52.4144577Z ok (4.083s)
2022-09-27T16:16:52.4144790Z 
2022-09-27T16:16:52.4145416Z ----------------------------------------------------------------------
2022-09-27T16:16:52.4145787Z Ran 1 test in 4.084s
2022-09-27T16:16:52.4145955Z 
2022-09-27T16:16:52.4146049Z OK
2022-09-27T16:16:52.4146184Z 
2022-09-27T16:16:52.4146335Z Generating XML reports...
2022-09-27T16:16:52.4181269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161648.xml
2022-09-27T16:16:54.3669138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:54.3669642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:54.3671883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:54.3672361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:54.8749413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:54.8774077Z 
2022-09-27T16:16:54.8774480Z Running tests...
2022-09-27T16:16:54.8774924Z ----------------------------------------------------------------------
2022-09-27T16:16:54.8800792Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:16:54.8801666Z 
2022-09-27T16:16:54.8802028Z ----------------------------------------------------------------------
2022-09-27T16:16:54.8802356Z Ran 1 test in 0.003s
2022-09-27T16:16:54.8802504Z 
2022-09-27T16:16:54.8802613Z OK (skipped=1)
2022-09-27T16:16:54.8802767Z 
2022-09-27T16:16:54.8802894Z Generating XML reports...
2022-09-27T16:16:54.8838131Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161654.xml
2022-09-27T16:16:56.7218011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:56.7218528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:56.7219493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:56.7219962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:57.2268808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:57.2291915Z 
2022-09-27T16:16:57.2292466Z Running tests...
2022-09-27T16:16:57.2292993Z ----------------------------------------------------------------------
2022-09-27T16:16:57.2313896Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:16:57.2314419Z 
2022-09-27T16:16:57.2314828Z ----------------------------------------------------------------------
2022-09-27T16:16:57.2315164Z Ran 1 test in 0.002s
2022-09-27T16:16:57.2315333Z 
2022-09-27T16:16:57.2315429Z OK (skipped=1)
2022-09-27T16:16:57.2315587Z 
2022-09-27T16:16:57.2315713Z Generating XML reports...
2022-09-27T16:16:57.2349043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161657.xml
2022-09-27T16:16:59.0956761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:16:59.0957345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:16:59.0958714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:16:59.0959218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:16:59.5908821Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:16:59.5931033Z 
2022-09-27T16:16:59.5931267Z Running tests...
2022-09-27T16:16:59.5931815Z ----------------------------------------------------------------------
2022-09-27T16:17:01.1504952Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:01.1694516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62623
2022-09-27T16:17:01.1700630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62624
2022-09-27T16:17:02.7726225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:02.7727193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:02.7728409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:02.7729382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:02.8049304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:02.8050235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:02.8054081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:02.8055051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:03.3236995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:03.3237543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:03.3238328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:03.3239014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:03.3345104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:03.3410962Z STAGE:2022-09-27 16:17:03 62624:62624 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:03.4249923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:03.4313605Z STAGE:2022-09-27 16:17:03 62623:62623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:03.4332123Z STAGE:2022-09-27 16:17:03 62623:62623 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:17:03 62624:62624 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:03.4332529Z 
2022-09-27T16:17:03.4350717Z STAGE:2022-09-27 16:17:03 62624:62624 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:03.4359646Z STAGE:2022-09-27 16:17:03 62623:62623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:03.4376964Z STAGE:2022-09-27 16:17:03 62623:62623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:03.4377549Z STAGE:2022-09-27 16:17:03 62624:62624 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:03.7765519Z ok (4.183s)
2022-09-27T16:17:03.7765737Z 
2022-09-27T16:17:03.7766103Z ----------------------------------------------------------------------
2022-09-27T16:17:03.7766459Z Ran 1 test in 4.183s
2022-09-27T16:17:03.7766624Z 
2022-09-27T16:17:03.7766702Z OK
2022-09-27T16:17:03.7766834Z 
2022-09-27T16:17:03.7766968Z Generating XML reports...
2022-09-27T16:17:03.7801861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161659.xml
2022-09-27T16:17:05.7639356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:05.7639999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:05.7641025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:05.7641768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:06.2576585Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:06.2598050Z 
2022-09-27T16:17:06.2598362Z Running tests...
2022-09-27T16:17:06.2599063Z ----------------------------------------------------------------------
2022-09-27T16:17:07.8311366Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:07.8574346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62736
2022-09-27T16:17:07.8580742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62737
2022-09-27T16:17:09.4974055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:09.4974576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:09.4976473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:09.4976991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:09.5293343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:09.5294173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:09.5297661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:09.5298164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:10.0518050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:10.0518608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:10.0519375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:10.0520095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:10.0526813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:10.0527315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:10.0624085Z STAGE:2022-09-27 16:17:10 62736:62736 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:10.0624689Z STAGE:2022-09-27 16:17:10 62737:62737 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:10.0638443Z STAGE:2022-09-27 16:17:10 62737:62737 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:10.0639034Z STAGE:2022-09-27 16:17:10 62736:62736 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:10.0659791Z STAGE:2022-09-27 16:17:10 62736:62736 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:10.0660629Z STAGE:2022-09-27 16:17:10 62737:62737 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:10.0672125Z STAGE:2022-09-27 16:17:10 62737:62737 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:10.0672728Z STAGE:2022-09-27 16:17:10 62736:62736 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:10.4643094Z ok (4.204s)
2022-09-27T16:17:10.4643315Z 
2022-09-27T16:17:10.4643699Z ----------------------------------------------------------------------
2022-09-27T16:17:10.4644028Z Ran 1 test in 4.204s
2022-09-27T16:17:10.4644196Z 
2022-09-27T16:17:10.4644292Z OK
2022-09-27T16:17:10.4644434Z 
2022-09-27T16:17:10.4644572Z Generating XML reports...
2022-09-27T16:17:10.4679804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161706.xml
2022-09-27T16:17:12.4441578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:12.4442128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:12.4443079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:12.4443588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:12.9355549Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:12.9377638Z 
2022-09-27T16:17:12.9378103Z Running tests...
2022-09-27T16:17:12.9378614Z ----------------------------------------------------------------------
2022-09-27T16:17:14.5039830Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:14.5229039Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62849
2022-09-27T16:17:14.5240476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62850
2022-09-27T16:17:16.0890545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:16.0891064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:16.0892278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:16.0892754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:16.1160711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:16.1161387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:16.1164929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:16.1165399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:16.6438069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:16.6438633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:16.6439459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:16.6440412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:16.6548315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:16.7448182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:17.1296058Z ok (4.192s)
2022-09-27T16:17:17.1296262Z 
2022-09-27T16:17:17.1296640Z ----------------------------------------------------------------------
2022-09-27T16:17:17.1296993Z Ran 1 test in 4.192s
2022-09-27T16:17:17.1297159Z 
2022-09-27T16:17:17.1297253Z OK
2022-09-27T16:17:17.1297405Z 
2022-09-27T16:17:17.1297540Z Generating XML reports...
2022-09-27T16:17:17.1332544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161712.xml
2022-09-27T16:17:19.0894378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:19.0895085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:19.0896959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:19.0897435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:19.5816489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:19.5838195Z 
2022-09-27T16:17:19.5838456Z Running tests...
2022-09-27T16:17:19.5839066Z ----------------------------------------------------------------------
2022-09-27T16:17:21.1463269Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:21.1654204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62958
2022-09-27T16:17:21.1660870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62959
2022-09-27T16:17:22.7346743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:22.7347273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:22.7348308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:22.7348790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:22.7626807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:22.7627314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:22.7630626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:22.7631459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:23.2879980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:23.2881038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:23.2881856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:23.2882570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:23.2987758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:23.3053614Z STAGE:2022-09-27 16:17:23 62959:62959 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:23.3891116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:23.3955660Z STAGE:2022-09-27 16:17:23 62958:62958 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:23.3969147Z STAGE:2022-09-27 16:17:23 62958:62958 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:23.3969758Z STAGE:2022-09-27 16:17:23 62959:62959 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:23.3990340Z STAGE:2022-09-27 16:17:23 62958:62958 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:17:23 62959:62959 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:23.3990740Z 
2022-09-27T16:17:23.4003480Z STAGE:2022-09-27 16:17:23 62958:62958 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:17:23 62959:62959 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:23.4003885Z 
2022-09-27T16:17:23.7723985Z ok (4.188s)
2022-09-27T16:17:23.7724226Z 
2022-09-27T16:17:23.7724650Z ----------------------------------------------------------------------
2022-09-27T16:17:23.7724976Z Ran 1 test in 4.188s
2022-09-27T16:17:23.7725152Z 
2022-09-27T16:17:23.7725248Z OK
2022-09-27T16:17:23.7726506Z 
2022-09-27T16:17:23.7726938Z Generating XML reports...
2022-09-27T16:17:23.7760949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161719.xml
2022-09-27T16:17:25.7730743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:25.7731756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:25.7732966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:25.7734298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:26.2634768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:26.2656937Z 
2022-09-27T16:17:26.2657202Z Running tests...
2022-09-27T16:17:26.2657646Z ----------------------------------------------------------------------
2022-09-27T16:17:26.2676486Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:17:26.2676794Z 
2022-09-27T16:17:26.2677079Z ----------------------------------------------------------------------
2022-09-27T16:17:26.2677409Z Ran 1 test in 0.002s
2022-09-27T16:17:26.2677575Z 
2022-09-27T16:17:26.2677672Z OK (skipped=1)
2022-09-27T16:17:26.2677832Z 
2022-09-27T16:17:26.2677958Z Generating XML reports...
2022-09-27T16:17:26.2711830Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161726.xml
2022-09-27T16:17:28.0992324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:28.0992826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:28.0994459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:28.0994951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:28.6105136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:28.6128385Z 
2022-09-27T16:17:28.6128646Z Running tests...
2022-09-27T16:17:28.6129084Z ----------------------------------------------------------------------
2022-09-27T16:17:28.6148447Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:17:28.6148746Z 
2022-09-27T16:17:28.6149044Z ----------------------------------------------------------------------
2022-09-27T16:17:28.6149381Z Ran 1 test in 0.002s
2022-09-27T16:17:28.6149551Z 
2022-09-27T16:17:28.6149666Z OK (skipped=1)
2022-09-27T16:17:28.6149824Z 
2022-09-27T16:17:28.6149961Z Generating XML reports...
2022-09-27T16:17:28.6184865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161728.xml
2022-09-27T16:17:30.4545374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:30.4545895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:30.4547587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:30.4548064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:30.9633179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:30.9656639Z 
2022-09-27T16:17:30.9656985Z Running tests...
2022-09-27T16:17:30.9657437Z ----------------------------------------------------------------------
2022-09-27T16:17:32.5320716Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:32.5516653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63137
2022-09-27T16:17:32.5523165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63138
2022-09-27T16:17:34.1256186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:34.1256708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:34.1258908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:34.1259388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:34.1493553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:34.1494053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:34.1497261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:34.1497740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:34.6742124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:34.6742677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:34.6743460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:34.6744148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:34.6750702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:34.6751207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:34.6859283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:17:34.6859797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:17:34.6860492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:17:34.6861333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:17:34.6954230Z STAGE:2022-09-27 16:17:34 63138:63138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:34.6954830Z STAGE:2022-09-27 16:17:34 63137:63137 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:34.6968783Z STAGE:2022-09-27 16:17:34 63138:63138 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:17:34 63137:63137 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:34.6969202Z 
2022-09-27T16:17:34.6988563Z STAGE:2022-09-27 16:17:34 63137:63137 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:17:34 63138:63138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:17:34.6988965Z 
2022-09-27T16:17:34.7001895Z STAGE:2022-09-27 16:17:34 63137:63137 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:34.7002510Z STAGE:2022-09-27 16:17:34 63138:63138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:17:35.1585668Z ok (4.193s)
2022-09-27T16:17:35.1585939Z 
2022-09-27T16:17:35.1586343Z ----------------------------------------------------------------------
2022-09-27T16:17:35.1586694Z Ran 1 test in 4.193s
2022-09-27T16:17:35.1586888Z 
2022-09-27T16:17:35.1586984Z OK
2022-09-27T16:17:35.1587103Z 
2022-09-27T16:17:35.1587244Z Generating XML reports...
2022-09-27T16:17:35.1623072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161730.xml
2022-09-27T16:17:37.1303905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:37.1304999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:37.1306783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:37.1307767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:37.6457457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:37.6482324Z 
2022-09-27T16:17:37.6482772Z Running tests...
2022-09-27T16:17:37.6483503Z ----------------------------------------------------------------------
2022-09-27T16:17:39.2344608Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:39.2536332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63256
2022-09-27T16:17:39.2543671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63257
2022-09-27T16:17:40.8283975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:40.8284452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:40.8287217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:40.8287719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:40.8567922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:40.8568378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:40.8572798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:40.8573500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:41.3793100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:41.3794269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:41.3795333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:41.3893612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:41.3905565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:41.4910455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:41.8604964Z skip: Skipped due to small world size. (4.212s)
2022-09-27T16:17:41.8605215Z 
2022-09-27T16:17:41.8605915Z ----------------------------------------------------------------------
2022-09-27T16:17:41.8606295Z Ran 1 test in 4.212s
2022-09-27T16:17:41.8606468Z 
2022-09-27T16:17:41.8606565Z OK (skipped=1)
2022-09-27T16:17:41.8606734Z 
2022-09-27T16:17:41.8606862Z Generating XML reports...
2022-09-27T16:17:41.8642723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161737.xml
2022-09-27T16:17:43.8504241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:43.8504735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:43.8506041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:43.8506540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:44.3605407Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:44.3628665Z 
2022-09-27T16:17:44.3629011Z Running tests...
2022-09-27T16:17:44.3629459Z ----------------------------------------------------------------------
2022-09-27T16:17:45.9549425Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:45.9750456Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63365
2022-09-27T16:17:45.9756311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63366
2022-09-27T16:17:47.5419329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:47.5420176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:47.5421350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:47.5421851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:47.5699540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:47.5700015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:47.5703968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:47.5704461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:48.0936416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:48.0936982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:48.0937786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:48.0938722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:48.0944999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:48.0945923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:48.4818184Z ok (4.119s)
2022-09-27T16:17:48.4818399Z 
2022-09-27T16:17:48.4818801Z ----------------------------------------------------------------------
2022-09-27T16:17:48.4819143Z Ran 1 test in 4.119s
2022-09-27T16:17:48.4819318Z 
2022-09-27T16:17:48.4819418Z OK
2022-09-27T16:17:48.4819539Z 
2022-09-27T16:17:48.4819675Z Generating XML reports...
2022-09-27T16:17:48.4855544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161744.xml
2022-09-27T16:17:50.4505075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:50.4505597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:50.4506783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:50.4507275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:50.9529998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:50.9552304Z 
2022-09-27T16:17:50.9552550Z Running tests...
2022-09-27T16:17:50.9552972Z ----------------------------------------------------------------------
2022-09-27T16:17:52.5005589Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:52.5195129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63474
2022-09-27T16:17:52.5201497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63475
2022-09-27T16:17:54.0994626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:54.0995173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:54.0996398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:54.0996940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:54.1201673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:54.1202343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:54.1206191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:54.1206840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:54.6480546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:17:54.6481148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:17:54.6481938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:54.6482623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:17:54.6488614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:17:54.6489386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:17:55.0292416Z ok (4.074s)
2022-09-27T16:17:55.0292761Z 
2022-09-27T16:17:55.0293411Z ----------------------------------------------------------------------
2022-09-27T16:17:55.0293745Z Ran 1 test in 4.074s
2022-09-27T16:17:55.0293920Z 
2022-09-27T16:17:55.0294018Z OK
2022-09-27T16:17:55.0294447Z 
2022-09-27T16:17:55.0294593Z Generating XML reports...
2022-09-27T16:17:55.0329936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161750.xml
2022-09-27T16:17:57.0495329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:17:57.0495857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:17:57.0497353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:17:57.0497850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:17:57.5599702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:17:57.5623331Z 
2022-09-27T16:17:57.5623562Z Running tests...
2022-09-27T16:17:57.5624004Z ----------------------------------------------------------------------
2022-09-27T16:17:59.1490210Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:17:59.1686574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63583
2022-09-27T16:17:59.1693192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63584
2022-09-27T16:18:00.7433571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:00.7434088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:00.7435277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:00.7435773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:00.7724691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:00.7725177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:00.7728304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:00.7728798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:01.3114349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:01.3114904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:01.3115703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:01.3116631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:01.3122893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:01.3123904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:01.6754071Z ok (4.113s)
2022-09-27T16:18:01.6754380Z 
2022-09-27T16:18:01.6754948Z ----------------------------------------------------------------------
2022-09-27T16:18:01.6755284Z Ran 1 test in 4.113s
2022-09-27T16:18:01.6755455Z 
2022-09-27T16:18:01.6755554Z OK
2022-09-27T16:18:01.6755698Z 
2022-09-27T16:18:01.6755834Z Generating XML reports...
2022-09-27T16:18:01.6791685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161757.xml
2022-09-27T16:18:03.6971334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:03.6971853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:03.6974629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:03.6975111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:04.2080781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:04.2104698Z 
2022-09-27T16:18:04.2104900Z Running tests...
2022-09-27T16:18:04.2105359Z ----------------------------------------------------------------------
2022-09-27T16:18:05.7702203Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:05.7893073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63692
2022-09-27T16:18:05.7900075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63693
2022-09-27T16:18:07.3756848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:07.3757782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:07.3758666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:07.3759172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:07.4730353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:07.4730851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:07.4733719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:07.4734247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:07.9848149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:07.9848768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:07.9849562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:07.9850290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:07.9856555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:07.9857034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:07.9953240Z STAGE:2022-09-27 16:18:07 63692:63692 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:07.9953839Z STAGE:2022-09-27 16:18:07 63693:63693 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:07.9977873Z STAGE:2022-09-27 16:18:07 63692:63692 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:07.9978713Z STAGE:2022-09-27 16:18:07 63693:63693 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:08.3960391Z ok (4.185s)
2022-09-27T16:18:08.3960605Z 
2022-09-27T16:18:08.3961216Z ----------------------------------------------------------------------
2022-09-27T16:18:08.3961572Z Ran 1 test in 4.185s
2022-09-27T16:18:08.3961741Z 
2022-09-27T16:18:08.3961844Z OK
2022-09-27T16:18:08.3961981Z 
2022-09-27T16:18:08.3962099Z Generating XML reports...
2022-09-27T16:18:08.3997376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161804.xml
2022-09-27T16:18:10.3448598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:10.3449080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:10.3450313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:10.3450805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:10.8332843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:10.8354800Z 
2022-09-27T16:18:10.8355001Z Running tests...
2022-09-27T16:18:10.8355442Z ----------------------------------------------------------------------
2022-09-27T16:18:12.3778801Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:12.3969683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63805
2022-09-27T16:18:12.3976815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63806
2022-09-27T16:18:13.9840781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:13.9841803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:13.9842982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:13.9844154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:14.0095132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:14.0096005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:14.0097764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:14.0098642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:14.5464475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:14.5465067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:14.5465968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:14.5466678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:14.5572408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:14.5638009Z STAGE:2022-09-27 16:18:14 63806:63806 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:14.6474901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:14.6539266Z STAGE:2022-09-27 16:18:14 63805:63805 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:14.6563670Z STAGE:2022-09-27 16:18:14 63805:63805 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:14.6565181Z STAGE:2022-09-27 16:18:14 63806:63806 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:15.0036816Z ok (4.168s)
2022-09-27T16:18:15.0037014Z 
2022-09-27T16:18:15.0037406Z ----------------------------------------------------------------------
2022-09-27T16:18:15.0037784Z Ran 1 test in 4.168s
2022-09-27T16:18:15.0037953Z 
2022-09-27T16:18:15.0038050Z OK
2022-09-27T16:18:15.0038189Z 
2022-09-27T16:18:15.0038332Z Generating XML reports...
2022-09-27T16:18:15.0074293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161810.xml
2022-09-27T16:18:17.0228037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:17.0228546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:17.0229997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:17.0230488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:17.5341988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:17.5365988Z 
2022-09-27T16:18:17.5366409Z Running tests...
2022-09-27T16:18:17.5366840Z ----------------------------------------------------------------------
2022-09-27T16:18:19.0979747Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:19.1177471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63918
2022-09-27T16:18:19.1184251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63919
2022-09-27T16:18:20.7521118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:20.7521655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:20.7522730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:20.7523208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:20.7666311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:20.7666811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:20.7670830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:20.7671305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:21.3009052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:21.3009659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:21.3010472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:21.3011187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:21.3117950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:21.3182315Z STAGE:2022-09-27 16:18:21 63919:63919 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:21.4021471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:21.4085107Z STAGE:2022-09-27 16:18:21 63918:63918 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:21.4107862Z STAGE:2022-09-27 16:18:21 63919:63919 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:21.4116505Z STAGE:2022-09-27 16:18:21 63918:63918 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:21.8247856Z ok (4.288s)
2022-09-27T16:18:21.8248043Z 
2022-09-27T16:18:21.8248592Z ----------------------------------------------------------------------
2022-09-27T16:18:21.8248971Z Ran 1 test in 4.288s
2022-09-27T16:18:21.8249143Z 
2022-09-27T16:18:21.8249240Z OK
2022-09-27T16:18:21.8249389Z 
2022-09-27T16:18:21.8249527Z Generating XML reports...
2022-09-27T16:18:21.8284841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161817.xml
2022-09-27T16:18:23.8046398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:23.8046919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:23.8049654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:23.8050130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:24.3207448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:24.3230582Z 
2022-09-27T16:18:24.3231040Z Running tests...
2022-09-27T16:18:24.3231548Z ----------------------------------------------------------------------
2022-09-27T16:18:24.3249600Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:18:24.3250113Z 
2022-09-27T16:18:24.3250454Z ----------------------------------------------------------------------
2022-09-27T16:18:24.3250779Z Ran 1 test in 0.002s
2022-09-27T16:18:24.3251048Z 
2022-09-27T16:18:24.3251159Z OK (skipped=1)
2022-09-27T16:18:24.3251321Z 
2022-09-27T16:18:24.3251449Z Generating XML reports...
2022-09-27T16:18:24.3285901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161824.xml
2022-09-27T16:18:26.1608850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:26.1609384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:26.1610736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:26.1611236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:26.6491406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:26.6513146Z 
2022-09-27T16:18:26.6513413Z Running tests...
2022-09-27T16:18:26.6513856Z ----------------------------------------------------------------------
2022-09-27T16:18:26.6531927Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:18:26.6532242Z 
2022-09-27T16:18:26.6532524Z ----------------------------------------------------------------------
2022-09-27T16:18:26.6532864Z Ran 1 test in 0.002s
2022-09-27T16:18:26.6533031Z 
2022-09-27T16:18:26.6533149Z OK (skipped=1)
2022-09-27T16:18:26.6533290Z 
2022-09-27T16:18:26.6533434Z Generating XML reports...
2022-09-27T16:18:26.6566070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161826.xml
2022-09-27T16:18:28.4945898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:28.4946409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:28.4947317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:28.4947806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:28.9983508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:29.0006015Z 
2022-09-27T16:18:29.0006274Z Running tests...
2022-09-27T16:18:29.0006726Z ----------------------------------------------------------------------
2022-09-27T16:18:29.0026300Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:18:29.0026645Z 
2022-09-27T16:18:29.0026945Z ----------------------------------------------------------------------
2022-09-27T16:18:29.0027317Z Ran 1 test in 0.002s
2022-09-27T16:18:29.0027487Z 
2022-09-27T16:18:29.0027600Z OK (skipped=1)
2022-09-27T16:18:29.0027758Z 
2022-09-27T16:18:29.0027867Z Generating XML reports...
2022-09-27T16:18:29.0060663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161828.xml
2022-09-27T16:18:30.8148441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:30.8148921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:30.8150936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:30.8151447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:31.3151095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:31.3183792Z 
2022-09-27T16:18:31.3184312Z Running tests...
2022-09-27T16:18:31.3184798Z ----------------------------------------------------------------------
2022-09-27T16:18:32.8859556Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:32.9053514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64130
2022-09-27T16:18:32.9059165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64131
2022-09-27T16:18:34.4568292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:34.4568805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:34.4570096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:34.4570593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:34.5217792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:34.5218297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:34.5220349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:34.5220847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:35.0479627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:35.0480461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:35.0481258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:35.0481972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:35.0488228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:35.0488724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:35.0583886Z STAGE:2022-09-27 16:18:35 64130:64130 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:35.0584489Z STAGE:2022-09-27 16:18:35 64131:64131 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:35.0608355Z STAGE:2022-09-27 16:18:35 64131:64131 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:35.0616574Z STAGE:2022-09-27 16:18:35 64130:64130 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:35.4125878Z ok (4.094s)
2022-09-27T16:18:35.4126066Z 
2022-09-27T16:18:35.4126675Z ----------------------------------------------------------------------
2022-09-27T16:18:35.4127038Z Ran 1 test in 4.094s
2022-09-27T16:18:35.4127211Z 
2022-09-27T16:18:35.4127302Z OK
2022-09-27T16:18:35.4127441Z 
2022-09-27T16:18:35.4127573Z Generating XML reports...
2022-09-27T16:18:35.4162998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161831.xml
2022-09-27T16:18:37.4049018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:37.4049533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:37.4050605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:37.4051103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:37.9200915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:37.9224657Z 
2022-09-27T16:18:37.9225058Z Running tests...
2022-09-27T16:18:37.9225579Z ----------------------------------------------------------------------
2022-09-27T16:18:39.4844557Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:39.5042585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64243
2022-09-27T16:18:39.5049442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64244
2022-09-27T16:18:41.1022471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:41.1022976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:41.1025154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:41.1025646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:41.1810525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:41.1811037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:41.1813091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:41.1813563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:41.7106586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:41.7107153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:41.7107945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:41.7108664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:41.7115580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:41.7116095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:42.1157414Z ok (4.193s)
2022-09-27T16:18:42.1157644Z 
2022-09-27T16:18:42.1158044Z ----------------------------------------------------------------------
2022-09-27T16:18:42.1158371Z Ran 1 test in 4.193s
2022-09-27T16:18:42.1158537Z 
2022-09-27T16:18:42.1158631Z OK
2022-09-27T16:18:42.1160020Z 
2022-09-27T16:18:42.1160669Z Generating XML reports...
2022-09-27T16:18:42.1252277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161837.xml
2022-09-27T16:18:44.1226513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:44.1227399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:44.1228327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:44.1228814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:44.6366217Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:44.6389608Z 
2022-09-27T16:18:44.6389992Z Running tests...
2022-09-27T16:18:44.6390512Z ----------------------------------------------------------------------
2022-09-27T16:18:46.2026507Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:46.2222597Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64352
2022-09-27T16:18:46.2228928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64353
2022-09-27T16:18:47.7997436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:47.7997978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:47.7998951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:47.7999742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:47.8214220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:47.8214693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:47.8218168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:47.8218667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:48.3446167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:48.3447105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:48.3447906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:48.3448613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:48.3454502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:48.3455561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:48.3549269Z STAGE:2022-09-27 16:18:48 64353:64353 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:48.3550213Z STAGE:2022-09-27 16:18:48 64352:64352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:48.3565987Z STAGE:2022-09-27 16:18:48 64352:64352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:48.3568067Z STAGE:2022-09-27 16:18:48 64353:64353 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:48.7291067Z ok (4.090s)
2022-09-27T16:18:48.7291413Z 
2022-09-27T16:18:48.7292163Z ----------------------------------------------------------------------
2022-09-27T16:18:48.7292827Z Ran 1 test in 4.090s
2022-09-27T16:18:48.7293023Z 
2022-09-27T16:18:48.7293121Z OK
2022-09-27T16:18:48.7293260Z 
2022-09-27T16:18:48.7295937Z Generating XML reports...
2022-09-27T16:18:48.7329320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161844.xml
2022-09-27T16:18:50.7200647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:50.7201187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:50.7202626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:50.7203157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:51.2148904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:51.2171808Z 
2022-09-27T16:18:51.2172023Z Running tests...
2022-09-27T16:18:51.2172521Z ----------------------------------------------------------------------
2022-09-27T16:18:52.7497746Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:52.7689868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64465
2022-09-27T16:18:52.7696572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64466
2022-09-27T16:18:54.3332939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:54.3333482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:54.3335612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:54.3336389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:54.3795453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:54.3795929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:54.3799209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:54.3799694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:54.8871649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:18:54.8872202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:18:54.8873014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:54.8873727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:18:54.8981178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:18:54.9047057Z STAGE:2022-09-27 16:18:54 64466:64466 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:54.9882012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:18:54.9946480Z STAGE:2022-09-27 16:18:54 64465:64465 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:18:54.9963080Z STAGE:2022-09-27 16:18:54 64465:64465 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:54.9965199Z STAGE:2022-09-27 16:18:54 64466:64466 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:18:55.3759204Z ok (4.158s)
2022-09-27T16:18:55.3759459Z 
2022-09-27T16:18:55.3760073Z ----------------------------------------------------------------------
2022-09-27T16:18:55.3760868Z Ran 1 test in 4.159s
2022-09-27T16:18:55.3761052Z 
2022-09-27T16:18:55.3761148Z OK
2022-09-27T16:18:55.3761288Z 
2022-09-27T16:18:55.3761407Z Generating XML reports...
2022-09-27T16:18:55.3796297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161851.xml
2022-09-27T16:18:57.3142053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:18:57.3142573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:18:57.3143885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:18:57.3144654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:18:57.8256784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:18:57.8280332Z 
2022-09-27T16:18:57.8280587Z Running tests...
2022-09-27T16:18:57.8281002Z ----------------------------------------------------------------------
2022-09-27T16:18:59.3720768Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:18:59.3918921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64578
2022-09-27T16:18:59.3925520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64579
2022-09-27T16:19:00.9696954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:00.9697461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:00.9699210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:00.9699707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:00.9879677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:00.9880741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:00.9884138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:00.9884631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:01.5196497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:01.5197062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:01.5197849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:01.5198557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:01.5304322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:01.6206716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:01.9987290Z ok (4.170s)
2022-09-27T16:19:01.9987525Z 
2022-09-27T16:19:01.9987916Z ----------------------------------------------------------------------
2022-09-27T16:19:01.9988268Z Ran 1 test in 4.171s
2022-09-27T16:19:01.9988436Z 
2022-09-27T16:19:01.9988515Z OK
2022-09-27T16:19:01.9988651Z 
2022-09-27T16:19:01.9988789Z Generating XML reports...
2022-09-27T16:19:02.0024362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161857.xml
2022-09-27T16:19:04.0320899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:04.0321407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:04.0322476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:04.0322967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:04.5265260Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:19:04.5288115Z 
2022-09-27T16:19:04.5288370Z Running tests...
2022-09-27T16:19:04.5288813Z ----------------------------------------------------------------------
2022-09-27T16:19:06.0772591Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:06.0967806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64777
2022-09-27T16:19:06.0974114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64778
2022-09-27T16:19:07.6941888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:07.6942424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:07.6943787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:07.6944257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:07.7378758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:07.7379229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:07.7382646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:07.7383120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:08.2676620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:08.2677167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:08.2678214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:08.2678905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:08.2684946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:08.2685436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:10.1060219Z ok (5.577s)
2022-09-27T16:19:10.1060419Z 
2022-09-27T16:19:10.1060822Z ----------------------------------------------------------------------
2022-09-27T16:19:10.1061163Z Ran 1 test in 5.577s
2022-09-27T16:19:10.1061351Z 
2022-09-27T16:19:10.1061450Z OK
2022-09-27T16:19:10.1061590Z 
2022-09-27T16:19:10.1061708Z Generating XML reports...
2022-09-27T16:19:10.1097615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161904.xml
2022-09-27T16:19:12.1066661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:12.1067148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:12.1068200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:12.1068688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:12.6198900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:19:12.6222490Z 
2022-09-27T16:19:12.6222805Z Running tests...
2022-09-27T16:19:12.6223259Z ----------------------------------------------------------------------
2022-09-27T16:19:14.1955447Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:14.2152174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64978
2022-09-27T16:19:14.2159028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64979
2022-09-27T16:19:15.7897303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:15.7897832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:15.7899908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:15.7900410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:15.8153979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:15.8154489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:15.8157412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:15.8157920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:16.3432582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:16.3433150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:16.3433934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:16.3434648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:16.3542493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:16.4447006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:18.7261050Z ok (6.104s)
2022-09-27T16:19:18.7261724Z 
2022-09-27T16:19:18.7262140Z ----------------------------------------------------------------------
2022-09-27T16:19:18.7262480Z Ran 1 test in 6.104s
2022-09-27T16:19:18.7262651Z 
2022-09-27T16:19:18.7262730Z OK
2022-09-27T16:19:18.7262867Z 
2022-09-27T16:19:18.7263008Z Generating XML reports...
2022-09-27T16:19:18.7298909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161912.xml
2022-09-27T16:19:20.7387769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:20.7388545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:20.7389458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:20.7389945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:21.2454176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:19:21.2477304Z 
2022-09-27T16:19:21.2477565Z Running tests...
2022-09-27T16:19:21.2477988Z ----------------------------------------------------------------------
2022-09-27T16:19:22.8295825Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:22.8493640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65093
2022-09-27T16:19:22.8500438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65094
2022-09-27T16:19:24.4270620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:24.4271157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:24.4273105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:24.4273597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:24.4533929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:24.4534418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:24.4538021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:24.4538558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:24.9802634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:24.9803166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:24.9804189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:24.9804927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:24.9911179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:25.0010361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq5k1sde
2022-09-27T16:19:25.0013556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq5k1sde/_remote_module_non_scriptable.py
2022-09-27T16:19:25.0814070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:25.0915796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd_mjh_e
2022-09-27T16:19:25.0918517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd_mjh_e/_remote_module_non_scriptable.py
2022-09-27T16:19:25.4573130Z ok (4.209s)
2022-09-27T16:19:25.4573337Z 
2022-09-27T16:19:25.4573727Z ----------------------------------------------------------------------
2022-09-27T16:19:25.4574070Z Ran 1 test in 4.209s
2022-09-27T16:19:25.4574498Z 
2022-09-27T16:19:25.4574577Z OK
2022-09-27T16:19:25.4574718Z 
2022-09-27T16:19:25.4574851Z Generating XML reports...
2022-09-27T16:19:25.4611160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161921.xml
2022-09-27T16:19:27.4898508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:27.4899005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:27.4901233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:27.4901726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:28.0009522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:19:28.0032194Z 
2022-09-27T16:19:28.0032448Z Running tests...
2022-09-27T16:19:28.0033131Z ----------------------------------------------------------------------
2022-09-27T16:19:29.5833223Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:29.6030336Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65206
2022-09-27T16:19:29.6037069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65207
2022-09-27T16:19:31.1747350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:31.1747904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:31.1750280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:31.1750806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:31.1981077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:31.1981600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:31.1984947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:31.1985447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:31.7238483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:31.7239548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:31.7240574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:31.7241512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:31.7346963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:31.8252601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:33.0330229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkft3zmz9
2022-09-27T16:19:33.0331413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkft3zmz9/_remote_module_non_scriptable.py
2022-09-27T16:19:33.1142058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s2uwds1
2022-09-27T16:19:33.1143494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s2uwds1/_remote_module_non_scriptable.py
2022-09-27T16:19:33.6143803Z ok (5.611s)
2022-09-27T16:19:33.6144193Z 
2022-09-27T16:19:33.6144930Z ----------------------------------------------------------------------
2022-09-27T16:19:33.6145629Z Ran 1 test in 5.611s
2022-09-27T16:19:33.6145868Z 
2022-09-27T16:19:33.6145966Z OK
2022-09-27T16:19:33.6146105Z 
2022-09-27T16:19:33.6146734Z Generating XML reports...
2022-09-27T16:19:33.6181872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161927.xml
2022-09-27T16:19:35.6336497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:35.6337104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:35.6339542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:35.6340053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:36.1397519Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:19:36.1421146Z 
2022-09-27T16:19:36.1421321Z Running tests...
2022-09-27T16:19:36.1421759Z ----------------------------------------------------------------------
2022-09-27T16:19:37.6863628Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:37.7210368Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65317
2022-09-27T16:19:37.7216254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65318
2022-09-27T16:19:39.2542642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:39.2543634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:39.2545333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:39.2546322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:39.3254771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:39.3255654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:39.3256983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:39.3257821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:39.8404203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:39.8404777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:39.8405568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:39.8406277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:39.8514010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:39.9420145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:41.1567383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph54id01n
2022-09-27T16:19:41.1568483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph54id01n/_remote_module_non_scriptable.py
2022-09-27T16:19:41.2432044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxosscn0p
2022-09-27T16:19:41.2433283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxosscn0p/_remote_module_non_scriptable.py
2022-09-27T16:19:41.6692725Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:19:41.6710891Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:19:42.1321401Z ok (5.990s)
2022-09-27T16:19:42.1321625Z 
2022-09-27T16:19:42.1322020Z ----------------------------------------------------------------------
2022-09-27T16:19:42.1322358Z Ran 1 test in 5.990s
2022-09-27T16:19:42.1322530Z 
2022-09-27T16:19:42.1322631Z OK
2022-09-27T16:19:42.1322768Z 
2022-09-27T16:19:42.1322903Z Generating XML reports...
2022-09-27T16:19:42.1357425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161936.xml
2022-09-27T16:19:44.1294524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:44.1295267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:44.1296822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:44.1297299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:44.6356034Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:19:44.6380371Z 
2022-09-27T16:19:44.6380630Z Running tests...
2022-09-27T16:19:44.6381089Z ----------------------------------------------------------------------
2022-09-27T16:19:46.1923032Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:46.2128197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65432
2022-09-27T16:19:46.2134308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65433
2022-09-27T16:19:47.8035140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:47.8035696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:47.8036868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:47.8037356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:47.8192137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:47.8192649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:47.8195746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:47.8196246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:48.3498309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:19:48.3498859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:19:48.3499660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:48.3500371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:19:48.3506450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:19:48.3506931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:19:48.3615154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:19:48.3615870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:19:48.3616565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:19:48.3617242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:19:48.3724038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:19:48.3724570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:19:48.3725280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:19:48.3725969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:19:49.6881120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8gvlbhr
2022-09-27T16:19:49.6881754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8gvlbhr/_remote_module_non_scriptable.py
2022-09-27T16:19:49.7119252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppgfvzpzp
2022-09-27T16:19:49.7122346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppgfvzpzp/_remote_module_non_scriptable.py
2022-09-27T16:19:55.2317964Z ok (10.593s)
2022-09-27T16:19:55.2318155Z 
2022-09-27T16:19:55.2318558Z ----------------------------------------------------------------------
2022-09-27T16:19:55.2318903Z Ran 1 test in 10.594s
2022-09-27T16:19:55.2319096Z 
2022-09-27T16:19:55.2319195Z OK
2022-09-27T16:19:55.2320576Z 
2022-09-27T16:19:55.2321389Z Generating XML reports...
2022-09-27T16:19:55.2355513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161944.xml
2022-09-27T16:19:57.2463640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:19:57.2464157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:19:57.2466774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:19:57.2467268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:19:57.7380599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:19:57.7403135Z 
2022-09-27T16:19:57.7403257Z Running tests...
2022-09-27T16:19:57.7404481Z ----------------------------------------------------------------------
2022-09-27T16:19:59.2700476Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:19:59.2891821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65555
2022-09-27T16:19:59.2898258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65556
2022-09-27T16:20:00.8747063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:00.8747579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:00.8748745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:00.8749236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:00.9035302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:00.9035795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:00.9039686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:00.9040625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:01.4263494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:01.4264049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:01.4264825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:01.4265536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:01.4272187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:01.4272685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:01.4380448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:20:01.4380995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:20:01.4381683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:20:01.4382381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:20:01.4690471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:20:01.4691005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:20:01.4691703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:20:01.4692401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:20:02.8120378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulrlkxvp
2022-09-27T16:20:02.8121021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulrlkxvp/_remote_module_non_scriptable.py
2022-09-27T16:20:02.8140347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_16tcof
2022-09-27T16:20:02.8142754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_16tcof/_remote_module_non_scriptable.py
2022-09-27T16:20:08.3085962Z ok (10.568s)
2022-09-27T16:20:08.3086249Z 
2022-09-27T16:20:08.3086657Z ----------------------------------------------------------------------
2022-09-27T16:20:08.3087003Z Ran 1 test in 10.568s
2022-09-27T16:20:08.3087169Z 
2022-09-27T16:20:08.3087514Z OK
2022-09-27T16:20:08.3087650Z 
2022-09-27T16:20:08.3087790Z Generating XML reports...
2022-09-27T16:20:08.3122865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161957.xml
2022-09-27T16:20:08.9660151Z Running distributed tests for the test backend with file init_method
2022-09-27T16:20:08.9678251Z Running distributed tests for the nccl backend with file init_method
2022-09-27T16:20:08.9686510Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:08.968184]
2022-09-27T16:20:08.9691190Z Running distributed tests for the gloo backend with file init_method
2022-09-27T16:20:08.9698256Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:08.969468]
2022-09-27T16:20:08.9704376Z Running distributed tests for the ucc backend with file init_method
2022-09-27T16:20:08.9711011Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:08.970731]
2022-09-27T16:20:08.9725495Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:20:08.972078]
2022-09-27T16:20:11.0261472Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:20:11.1126580Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:20:11.1201934Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-09-27T16:20:11.1262067Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1263312Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1264087Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1264883Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1265680Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1266541Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1267426Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1268346Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1269244Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1270386Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1271462Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1272430Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1273378Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1274254Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1275085Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1275894Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1276714Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1277460Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1278207Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1278990Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1279839Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1281034Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1281758Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1282448Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1283183Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1283906Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1284654Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1285440Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1286192Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1286854Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1287569Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1288268Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1288952Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1289634Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1290357Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1291105Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1291975Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1292711Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1293444Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1294222Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1295017Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1295821Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1296575Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1297289Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1298021Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1298759Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1299471Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1300262Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1301191Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1301922Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1302634Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1303353Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1304057Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1304700Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1305401Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1306087Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1306738Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1307397Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1308054Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1308730Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1309341Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1309956Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1310599Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1311252Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1311968Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1312681Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1313342Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1313998Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1314652Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1315442Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1316140Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1316892Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1317577Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1318202Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1318861Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1319547Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1320531Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1321261Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1321948Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1322741Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1323442Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1324195Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1324952Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1325725Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1326506Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1327281Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1328036Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1328836Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1329567Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1330321Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1331091Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1332032Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1332805Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1333523Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1334257Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1335024Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1335729Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1336390Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1337032Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1337648Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1338208Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1338831Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1339496Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1340202Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1340898Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1341597Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1342355Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1343032Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1343732Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1344456Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1345185Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1345912Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1346647Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1347403Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1348104Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1348860Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1349590Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1350265Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1350929Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1351600Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1352370Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1353072Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1353751Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1354428Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1355213Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1356114Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1356878Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1357622Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1358398Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1359185Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1360057Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1361353Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1362178Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1363173Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1363980Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1364756Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1365447Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1366154Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1366919Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1367766Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1368700Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1369548Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1370384Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1371191Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1371997Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1372954Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1373996Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1375127Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1376203Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1377314Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1378457Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1379580Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1380657Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1381754Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1382897Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1383798Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1384625Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1385346Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1386063Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1386816Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1387544Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1388319Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1389131Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1389958Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1390858Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1391633Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1392342Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1393196Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1393969Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1394764Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1395530Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1396300Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1397102Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1397845Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1398643Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1399412Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1400511Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1401316Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1402088Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1402895Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1403750Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1404562Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1405269Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1406004Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1406807Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1407596Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1408320Z test_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1408974Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1409673Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1410383Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1411072Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1411773Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1412505Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1413212Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1413900Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1414566Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1415346Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1416220Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1416989Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1417686Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1418336Z test_isend (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1419066Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1419828Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1420604Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1421473Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1422348Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1423140Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1423902Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1424739Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1425641Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1426405Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1427306Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1428075Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1428796Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1429543Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1430292Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1431061Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1431930Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1432842Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1433709Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1434508Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1435359Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1436176Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1436978Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1437743Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1438553Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1439372Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1440418Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1441348Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1442312Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1443184Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1443930Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1444660Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1445432Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1446188Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1446924Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1447647Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1448373Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1449207Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1449905Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1450551Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1451251Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1451952Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1452661Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1453355Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1454024Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1454682Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1455326Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1455980Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1456660Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1457397Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1458090Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1458839Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1459771Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1460486Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1461237Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1461933Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1462608Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1463377Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1464167Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1464972Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1465702Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1466414Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1467163Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1467881Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1468568Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1469289Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1470019Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1470793Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1471502Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1472277Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1472976Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1473720Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1474496Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1475280Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1476069Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1476752Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:20:11.1477320Z <unittest.suite.TestSuite tests=[]>
2022-09-27T16:20:11.1548162Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-09-27T16:20:11.1619180Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1686459Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-09-27T16:20:11.1749211Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1750197Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1751059Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1752038Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1752809Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1753556Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1754353Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1755151Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1755936Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1756816Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1757727Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1758658Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1759566Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1760781Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1761823Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1762785Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1763784Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1764781Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1765778Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1766722Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1767748Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1768724Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1769726Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1770687Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1771670Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1772647Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1773478Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1774353Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1775375Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1776315Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1777225Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1778062Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1778943Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1779830Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1780733Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1781593Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1782337Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1783121Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1783971Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1784758Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1785710Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1786486Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1787277Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1788069Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1788842Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1789629Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1790382Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1791179Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1791975Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1792707Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1793540Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1794362Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1795153Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1795936Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1796634Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1797337Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1798071Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1798873Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1799684Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1800743Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1801532Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1802281Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1803018Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1803773Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1804567Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1805376Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1806202Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1806997Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1807955Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1808788Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1809537Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1810282Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1811078Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1836409Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1837246Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1838043Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1838794Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1839590Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1841145Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1841699Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1842115Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1842713Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1843134Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1843544Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1843962Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1844405Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1844829Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1845232Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1845650Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1846080Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1846507Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1847187Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1847654Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1848048Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1848434Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1848858Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1849286Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1849698Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1850120Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1850505Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1850933Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1851335Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1851748Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1852175Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1852573Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1852998Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1853410Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1853825Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1854228Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1854617Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1855138Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1855561Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1855973Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1856395Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1856787Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1857190Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1857595Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1858043Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1858480Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1858896Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1859277Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1859662Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1860071Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1860473Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1860921Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1861339Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1861770Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1862172Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1862559Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1862969Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1863379Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1863760Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1864157Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1864569Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1864974Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1865396Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1865822Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1866262Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1866662Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1867092Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1867523Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1867930Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1868395Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1868833Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1869259Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1869665Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1870089Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1870529Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1870915Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1871335Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1871748Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1872157Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1872693Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1873135Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1873563Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1873991Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1874440Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1874875Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1875282Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1875869Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1876303Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1876723Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1877096Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1877491Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1877884Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1878327Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1878718Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1879096Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1879463Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1879828Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1880509Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1880926Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1881319Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1881710Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1882116Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1882507Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1882884Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1883277Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1883683Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1884062Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1884448Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1884839Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1885227Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1885630Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1886028Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1886423Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1886853Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1887271Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1887683Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1888091Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1888498Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1888894Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1889292Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1889702Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1890121Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1890536Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1891039Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1891486Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1891927Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1892301Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1892705Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1893114Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1893521Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1893952Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1894346Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1894771Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1895182Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1895616Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1896048Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1896574Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1897051Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1897537Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1898024Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1898441Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1898859Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1899286Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1899698Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1900133Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1900592Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1901027Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1901474Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1901940Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1902418Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1902836Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1903268Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1903729Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1904165Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1904579Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1905013Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1905430Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1905829Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1906263Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1906702Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1907131Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1907540Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1908004Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1908456Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1908871Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1909289Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1909691Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1910074Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1910491Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1910882Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1911308Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1911817Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1912255Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1912772Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1913385Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1913956Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1914472Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1915020Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1915526Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1916023Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1916536Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1917156Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1917646Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1918161Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1918671Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1919181Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1919736Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1920369Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1920849Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1921286Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1921692Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1922095Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1922472Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1922871Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1923292Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1923706Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1924112Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1924533Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1925013Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1925438Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1925883Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1926326Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1926775Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1927252Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1927683Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1928086Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1928465Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1928874Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1929305Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1929732Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1930179Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1930692Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1931112Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1931520Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1931932Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1932318Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1932702Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1933108Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1933518Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1933928Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1934348Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1934770Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1935174Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1935555Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1935955Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1936362Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1936760Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1937174Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1937577Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1938036Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1938513Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1939006Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1939505Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1939928Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1940328Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1940740Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1941126Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1941531Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1941948Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1942367Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1942827Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1943278Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1943685Z test_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1944039Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1944454Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1944863Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1945302Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1945726Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1946120Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1946505Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1946911Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1947327Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1947734Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1948228Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1948630Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1949006Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1949375Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1949727Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1950089Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1950478Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1950873Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1951287Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1951696Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1952075Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1952487Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1952871Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1953280Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1953671Z test_isend (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1954055Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1954474Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1954873Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1955280Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1955704Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1956149Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1956600Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1957036Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1957511Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1957948Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1958378Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1958855Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1959311Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1959903Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1960631Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1961168Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1961673Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1962199Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1962714Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1963241Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1963742Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1964275Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1964786Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1965386Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1965900Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1966424Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1966915Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1967305Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1967830Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1968391Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1968886Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1969368Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1969855Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1970308Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1970764Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1971256Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1971701Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1972117Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1972540Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1972951Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1973353Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1973760Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1974197Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1974639Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1975061Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1975482Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1975915Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1976362Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1976887Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1977400Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1977889Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1978333Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1978734Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1979182Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1979624Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1980016Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1980453Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1980880Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1981259Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1981653Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1982042Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1982496Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1982894Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1983315Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1983741Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1984137Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1984544Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1984944Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1985316Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1985708Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1986108Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1986512Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1986884Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1987296Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1987716Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1988116Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1988515Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1988911Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1989313Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1989686Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1990091Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1990486Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1990872Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1991265Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1991634Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1992019Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1992422Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1992807Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1993173Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1993562Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1993992Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1994418Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1994892Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1995330Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1995720Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1996095Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1996477Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1996845Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1997208Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1997620Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1998062Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1998511Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1998931Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1999357Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.1999779Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2000383Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2000804Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2001201Z test_gather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2001594Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2001977Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2002374Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2002762Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2003128Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2003518Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2003940Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2004336Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2004718Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2005122Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2005545Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2005931Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2006313Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2006708Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2007101Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2007481Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2007867Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2008240Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2008605Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2009018Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2009458Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2009872Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2010320Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2010742Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2011122Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2011494Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2011845Z test_isend (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2012211Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2012695Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2013126Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2013590Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2014032Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2014453Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2014925Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2015351Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2015782Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2016211Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2016633Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2017038Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2017452Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2017871Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2018338Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2018741Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2019210Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2019688Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2020168Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2020620Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2021086Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2021533Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2021969Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2022399Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2022815Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2023261Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2023718Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2024203Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2024704Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2025194Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2025620Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2026005Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2026410Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2026822Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2027214Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2027582Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2027974Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2028366Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2028722Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2029087Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2029467Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2029839Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2030285Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2030676Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2031058Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2031431Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2031818Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2032185Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2032539Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2032917Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2033293Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2033664Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2034061Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2034451Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2034821Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2035196Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2035641Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2036071Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2036507Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2036944Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2037341Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2037732Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2038161Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2038582Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2038986Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2039389Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2039832Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2040415Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2040816Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2041232Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2041637Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2042007Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2042434Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2042888Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:11.2043337Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-09-27T16:20:12.7113503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:12.7114242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:12.7115481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:12.7115993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:12.7255609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:12.7256091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:12.7260061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:12.7260571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:12.7438532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:12.7439039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:12.7442590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:12.7443093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:13.2293997Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:20:13.2317153Z 
2022-09-27T16:20:13.2317720Z Running tests...
2022-09-27T16:20:13.2318207Z ----------------------------------------------------------------------
2022-09-27T16:20:13.2361510Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:20:13.2384673Z 
2022-09-27T16:20:13.2385054Z Running tests...
2022-09-27T16:20:13.2385592Z ----------------------------------------------------------------------
2022-09-27T16:20:13.2704152Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:20:13.2727501Z 
2022-09-27T16:20:13.2727829Z Running tests...
2022-09-27T16:20:13.2728399Z ----------------------------------------------------------------------
2022-09-27T16:20:14.9480062Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:14.9480807Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:14.9487377Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:14.9704929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65888
2022-09-27T16:20:14.9705646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65889
2022-09-27T16:20:14.9706109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65890
2022-09-27T16:20:14.9711507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65891
2022-09-27T16:20:14.9712801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65892
2022-09-27T16:20:14.9713895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65893
2022-09-27T16:20:16.5920825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:16.5921388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:16.5923539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:16.5924039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:16.6045808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:16.6046300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:16.6049027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:16.6049518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:16.6091226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:16.6091676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:16.6096003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:16.6096493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:16.6151600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:16.6152076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:16.6155317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:16.6155810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:16.6463230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:16.6463696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:16.6466545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:16.6467032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:16.6478163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:16.6478609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:16.6482336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:16.6483062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:17.1198400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:17.1339601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:17.1340440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:17.1389961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:17.1400407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:17.1407743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:17.1825884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:17.1826717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:17.1902017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:17.1906822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:17.2003521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:17.2004115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:17.2004815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:17.2005521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:17.2114278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:17.2353539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:17.2841078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:17.3018156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:18.6087773Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:18.6089134Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:18.7047043Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:18.7048113Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:18.7150790Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:18.7151782Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:18.7641684Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:18.7642664Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:18.7787546Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:18.7788877Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:18.7867613Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:18.7868501Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.7647388Z [1664295619.764217] [28fd6574525b:65890:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:20:19.7647948Z [1664295619.764217] [28fd6574525b:65890:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:20:19.7687428Z [1664295619.768444] [28fd6574525b:65893:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:20:19.7687920Z [1664295619.768444] [28fd6574525b:65893:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:20:19.8670555Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.8671587Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.8672277Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.8673138Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.8818096Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.8819086Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.8819787Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.8820626Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.8949556Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.8950184Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.8951323Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.8952290Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.9634989Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.9635998Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.9638054Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.9639222Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.9761265Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.9762226Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.9764339Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.9765177Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.9887017Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.9887993Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:19.9889727Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:19.9890578Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:20.1094005Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:20.1095073Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:20.1095765Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:20.1096626Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:20.1209228Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:20.1210233Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:20.1210933Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:20.1211770Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:20.1326110Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:20.1327332Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:20.1328033Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-09-27T16:20:20.1328885Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-09-27T16:20:20.5831706Z ok (7.344s)
2022-09-27T16:20:20.5831950Z 
2022-09-27T16:20:20.5832354Z ----------------------------------------------------------------------
2022-09-27T16:20:20.5833202Z Ran 1 test in 7.345s
2022-09-27T16:20:20.5833516Z 
2022-09-27T16:20:20.5833701Z OK
2022-09-27T16:20:20.5833954Z 
2022-09-27T16:20:20.5834198Z Generating XML reports...
2022-09-27T16:20:20.5872878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162013.xml
2022-09-27T16:20:20.6834790Z ok (7.410s)
2022-09-27T16:20:20.6835130Z 
2022-09-27T16:20:20.6835800Z ----------------------------------------------------------------------
2022-09-27T16:20:20.6836444Z Ran 1 test in 7.410s
2022-09-27T16:20:20.6836755Z 
2022-09-27T16:20:20.6836931Z OK
2022-09-27T16:20:20.6837197Z 
2022-09-27T16:20:20.6837853Z Generating XML reports...
2022-09-27T16:20:20.6873418Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162013.xml
2022-09-27T16:20:20.7836757Z ok (7.551s)
2022-09-27T16:20:20.7837111Z 
2022-09-27T16:20:20.7837705Z ----------------------------------------------------------------------
2022-09-27T16:20:20.7838223Z Ran 1 test in 7.552s
2022-09-27T16:20:20.7838487Z 
2022-09-27T16:20:20.7838637Z OK
2022-09-27T16:20:20.7838861Z 
2022-09-27T16:20:20.7839068Z Generating XML reports...
2022-09-27T16:20:20.7874848Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162013.xml
2022-09-27T16:20:22.7409968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:22.7410487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:22.7412641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:22.7413153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:22.7900305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:22.7903756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:22.7904360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:22.7904839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:22.8940166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:22.8940670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:22.8943717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:22.8944208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:23.2209392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:20:23.2231899Z 
2022-09-27T16:20:23.2232049Z Running tests...
2022-09-27T16:20:23.2232495Z ----------------------------------------------------------------------
2022-09-27T16:20:23.2268317Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-09-27T16:20:23.2270089Z 
2022-09-27T16:20:23.2270517Z ----------------------------------------------------------------------
2022-09-27T16:20:23.2270872Z Ran 1 test in 0.004s
2022-09-27T16:20:23.2271038Z 
2022-09-27T16:20:23.2271435Z OK (skipped=1)
2022-09-27T16:20:23.2271612Z 
2022-09-27T16:20:23.2271742Z Generating XML reports...
2022-09-27T16:20:23.2304581Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162023.xml
2022-09-27T16:20:23.2726881Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:20:23.2750340Z 
2022-09-27T16:20:23.2750499Z Running tests...
2022-09-27T16:20:23.2751114Z ----------------------------------------------------------------------
2022-09-27T16:20:23.2789335Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-09-27T16:20:23.2790049Z 
2022-09-27T16:20:23.2790720Z ----------------------------------------------------------------------
2022-09-27T16:20:23.2791167Z Ran 1 test in 0.004s
2022-09-27T16:20:23.2791335Z 
2022-09-27T16:20:23.2791447Z OK (skipped=1)
2022-09-27T16:20:23.2791606Z 
2022-09-27T16:20:23.2791735Z Generating XML reports...
2022-09-27T16:20:23.2827438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162023.xml
2022-09-27T16:20:23.3968408Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:20:23.3991163Z 
2022-09-27T16:20:23.3991632Z Running tests...
2022-09-27T16:20:23.3992600Z ----------------------------------------------------------------------
2022-09-27T16:20:23.4033041Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-09-27T16:20:23.4033657Z 
2022-09-27T16:20:23.4034281Z ----------------------------------------------------------------------
2022-09-27T16:20:23.4034870Z Ran 1 test in 0.004s
2022-09-27T16:20:23.4035037Z 
2022-09-27T16:20:23.4035148Z OK (skipped=1)
2022-09-27T16:20:23.4035306Z 
2022-09-27T16:20:23.4035435Z Generating XML reports...
2022-09-27T16:20:23.4070648Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162023.xml
2022-09-27T16:20:25.2794951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:25.2795513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:25.2796322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:25.2796819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:25.3363192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:25.3363691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:25.3367254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:25.3367753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:25.4632354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:25.4632854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:25.4635192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:25.4635686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:25.7686769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:20:25.7708994Z 
2022-09-27T16:20:25.7709357Z Running tests...
2022-09-27T16:20:25.7709805Z ----------------------------------------------------------------------
2022-09-27T16:20:25.8336574Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:20:25.8359570Z 
2022-09-27T16:20:25.8360068Z Running tests...
2022-09-27T16:20:25.8360775Z ----------------------------------------------------------------------
2022-09-27T16:20:25.9540141Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:20:25.9562828Z 
2022-09-27T16:20:25.9563221Z Running tests...
2022-09-27T16:20:25.9563698Z ----------------------------------------------------------------------
2022-09-27T16:20:27.4045607Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:27.4240623Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66329
2022-09-27T16:20:27.4247688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66330
2022-09-27T16:20:27.4591390Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:27.4798734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66332
2022-09-27T16:20:27.4804718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66333
2022-09-27T16:20:27.5678920Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:27.5877094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66335
2022-09-27T16:20:27.5883182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66336
2022-09-27T16:20:29.2032490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:29.2033493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:29.2034691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:29.2035626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:29.3143294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:29.3144265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:29.3145440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:29.3146391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:29.3456217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:29.3457211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:29.3458732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:29.3459727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:29.3795234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:29.3796200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:29.3797359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:29.3798337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:29.4351507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:29.4352466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:29.4353655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:29.4354544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:29.4849381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:29.4850785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:29.4852022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:29.4852978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:29.8540272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:29.8541380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:29.8542894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:29.8544315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:29.8556274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:29.8648631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:29.9066867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:29.9068605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:29.9165588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:29.9171804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:29.9445640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:29.9550732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:29.9941902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:29.9942778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:29.9955121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:29.9964191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:30.0079628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:30.0955106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:30.4320920Z ok (4.661s)
2022-09-27T16:20:30.4321287Z 
2022-09-27T16:20:30.4321928Z ----------------------------------------------------------------------
2022-09-27T16:20:30.4322525Z Ran 1 test in 4.661s
2022-09-27T16:20:30.4322825Z 
2022-09-27T16:20:30.4322986Z OK
2022-09-27T16:20:30.4323232Z 
2022-09-27T16:20:30.4323497Z Generating XML reports...
2022-09-27T16:20:30.4358878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162025.xml
2022-09-27T16:20:30.4878193Z ok (4.652s)
2022-09-27T16:20:30.4878413Z 
2022-09-27T16:20:30.4878806Z ----------------------------------------------------------------------
2022-09-27T16:20:30.4879154Z Ran 1 test in 4.652s
2022-09-27T16:20:30.4879323Z 
2022-09-27T16:20:30.4879401Z OK
2022-09-27T16:20:30.4879539Z 
2022-09-27T16:20:30.4879681Z Generating XML reports...
2022-09-27T16:20:30.4916502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162025.xml
2022-09-27T16:20:30.4950478Z ok (4.539s)
2022-09-27T16:20:30.4950647Z 
2022-09-27T16:20:30.4950933Z ----------------------------------------------------------------------
2022-09-27T16:20:30.4951277Z Ran 1 test in 4.539s
2022-09-27T16:20:30.4951442Z 
2022-09-27T16:20:30.4951537Z OK
2022-09-27T16:20:30.4951675Z 
2022-09-27T16:20:30.4952036Z Generating XML reports...
2022-09-27T16:20:30.4989298Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162025.xml
2022-09-27T16:20:32.6638879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:32.6639935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:32.6641509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:32.6642486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:32.7224180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:32.7225061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:32.7228583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:32.7229576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:32.7235794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:32.7237091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:32.7241314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:32.7242301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:33.1820090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:20:33.1843672Z 
2022-09-27T16:20:33.1844145Z Running tests...
2022-09-27T16:20:33.1844653Z ----------------------------------------------------------------------
2022-09-27T16:20:33.2171924Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:20:33.2195462Z 
2022-09-27T16:20:33.2195710Z Running tests...
2022-09-27T16:20:33.2196138Z ----------------------------------------------------------------------
2022-09-27T16:20:33.2207547Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:20:33.2232073Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... 
2022-09-27T16:20:33.2232436Z Running tests...
2022-09-27T16:20:33.2233096Z ----------------------------------------------------------------------
2022-09-27T16:20:34.8587956Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:34.8631434Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:34.8662482Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:20:34.8784543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66648
2022-09-27T16:20:34.8790515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66649
2022-09-27T16:20:34.8829620Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66651
2022-09-27T16:20:34.8836561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66652
2022-09-27T16:20:34.8865537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66654
2022-09-27T16:20:34.8872467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66655
2022-09-27T16:20:36.4937998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:36.4938520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:36.4939099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:36.4939560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:36.4940426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:36.4940931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:36.4941553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:36.4942034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:36.4983762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:36.4984242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:36.4987669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:36.4988157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:36.5012584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:36.5013049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:36.5018480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:36.5018974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:36.5137788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:36.5138255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:36.5141782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:36.5142263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:36.5869902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:20:36.5870672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:20:36.5871265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:20:36.5871757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:20:37.0271387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:37.0347220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:37.0413629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:37.0414474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:37.0473246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:37.0479796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:37.0555317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:20:37.0555875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:37.0556593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:37.0557302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:37.0666536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:37.1092170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:20:37.1093837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:37.1159892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:20:37.1167734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:20:37.1427543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:37.1570424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:37.2106012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:20:38.5260421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8ycbvk7
2022-09-27T16:20:38.5261571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8ycbvk7/_remote_module_non_scriptable.py
2022-09-27T16:20:38.5453553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9a1j3ac
2022-09-27T16:20:38.5454448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9a1j3ac/_remote_module_non_scriptable.py
2022-09-27T16:20:38.5981420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuly9fy6
2022-09-27T16:20:38.5982375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuly9fy6/_remote_module_non_scriptable.py
2022-09-27T16:20:38.6569703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcmhxfhm
2022-09-27T16:20:38.6570686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcmhxfhm/_remote_module_non_scriptable.py
2022-09-27T16:20:38.6811994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfmy1vjv
2022-09-27T16:20:38.6813427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfmy1vjv/_remote_module_non_scriptable.py
2022-09-27T16:20:38.6936747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05j10lup
2022-09-27T16:20:38.6938155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05j10lup/_remote_module_non_scriptable.py
2022-09-27T16:20:38.7094930Z [1664295638.709059] [28fd6574525b:66651:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:20:38.7095436Z [1664295638.709059] [28fd6574525b:66651:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:20:38.7168908Z [1664295638.716468] [28fd6574525b:66652:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:20:38.7169425Z [1664295638.716468] [28fd6574525b:66652:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:20:39.1451317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.1451943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.1649407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.1649975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.1970165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.1970699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.3366486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.3367482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.3557612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.3558661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.3803079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.3804115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.3996529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.3997495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.4298679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.4299696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5074287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5074857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5244920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5245436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5528447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5529181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5594339Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T16:20:39.5595179Z   warnings.warn(
2022-09-27T16:20:39.5596335Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T16:20:39.5597154Z   warnings.warn(
2022-09-27T16:20:39.5774831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5775820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5902232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.5903201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.6084231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.6087973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.7316683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.7323091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.7392548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.7393525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.7524069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.8820506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.8821556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.8823573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.8865612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:39.8866590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:40.0113357Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T16:20:40.0114290Z   warnings.warn(
2022-09-27T16:20:40.0115765Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T16:20:40.0116671Z   warnings.warn(
2022-09-27T16:20:40.0266333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:40.0270477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:40.0663099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:40.0663627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:40.1065587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:40.1066126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:40.1286546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:20:40.1287596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:40.1340170Z [1664295700.133512] [28fd6574525b:66652:1]    ucc_schedule.h:182  UCC  WARN  timeout 60 sec has expired on req 0x556586bc0580, seq_num 104, TL_NCCL, team_id 1, size 2, rank 1, ctx_rank 1: Allgather Cuda inplace=0 bytes=16
2022-09-27T16:21:40.1432476Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-09-27T16:21:40.1435043Z Traceback (most recent call last):
2022-09-27T16:21:40.1435753Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:21:40.1436177Z     getattr(self, test_name)()
2022-09-27T16:21:40.1436708Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:21:40.1437071Z     fn()
2022-09-27T16:21:40.1439313Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T16:21:40.1439745Z     return func(*args, **kwargs)
2022-09-27T16:21:40.1440840Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4816, in test_DistributedDataParallel
2022-09-27T16:21:40.1441352Z     self._test_DistributedDataParallel(
2022-09-27T16:21:40.1441999Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4030, in _test_DistributedDataParallel
2022-09-27T16:21:40.1442452Z     self._test_DDP_niter(
2022-09-27T16:21:40.1443012Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3942, in _test_DDP_niter
2022-09-27T16:21:40.1443437Z     self._test_DDP_helper(
2022-09-27T16:21:40.1443981Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3906, in _test_DDP_helper
2022-09-27T16:21:40.1444931Z     l.backward()
2022-09-27T16:21:40.1445392Z   File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 482, in backward
2022-09-27T16:21:40.1445778Z     torch.autograd.backward(
2022-09-27T16:21:40.1446265Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
2022-09-27T16:21:40.1446755Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:21:40.1447315Z RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 60000ms for recv operation to complete
2022-09-27T16:21:40.1447753Z  exiting process 0 with exit code: 10
2022-09-27T16:21:40.4915314Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-09-27T16:21:40.4931107Z ERROR (67.309s)
2022-09-27T16:21:40.4932205Z     test_DistributedDataParallel errored - num_retries_left: 3
2022-09-27T16:21:40.4932962Z Traceback (most recent call last):
2022-09-27T16:21:40.4934194Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 501, in wrapper
2022-09-27T16:21:40.4935449Z     self._join_processes(fn)
2022-09-27T16:21:40.4936708Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 724, in _join_processes
2022-09-27T16:21:40.4937690Z     self._check_return_codes(elapsed_time)
2022-09-27T16:21:40.4938960Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 769, in _check_return_codes
2022-09-27T16:21:40.4939932Z     raise RuntimeError(error)
2022-09-27T16:21:40.4940662Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-09-27T16:21:40.4941428Z Traceback (most recent call last):
2022-09-27T16:21:40.4942632Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:21:40.4943496Z     getattr(self, test_name)()
2022-09-27T16:21:40.4944664Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:21:40.4945496Z     fn()
2022-09-27T16:21:40.4946594Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T16:21:40.4947446Z     return func(*args, **kwargs)
2022-09-27T16:21:40.4949019Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4816, in test_DistributedDataParallel
2022-09-27T16:21:40.4950092Z     self._test_DistributedDataParallel(
2022-09-27T16:21:40.4951473Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4030, in _test_DistributedDataParallel
2022-09-27T16:21:40.4952457Z     self._test_DDP_niter(
2022-09-27T16:21:40.4953703Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3942, in _test_DDP_niter
2022-09-27T16:21:40.4954635Z     self._test_DDP_helper(
2022-09-27T16:21:40.4955861Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3906, in _test_DDP_helper
2022-09-27T16:21:40.4956760Z     l.backward()
2022-09-27T16:21:40.4957722Z   File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 482, in backward
2022-09-27T16:21:40.4958519Z     torch.autograd.backward(
2022-09-27T16:21:40.4959600Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
2022-09-27T16:21:40.4961232Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:21:40.4962483Z RuntimeError: [/var/lib/jenkins/workspace/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 60000ms for recv operation to complete
2022-09-27T16:21:40.4963218Z 
2022-09-27T16:21:40.4963279Z 
2022-09-27T16:21:40.4963309Z 
2022-09-27T16:21:40.4964065Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66897
2022-09-27T16:21:40.4968326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66898
2022-09-27T16:21:45.6765428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:45.6766414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:45.6767613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:45.6768511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:45.6774404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:45.6775354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:45.6778651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:45.6779864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:46.2081800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:46.2082412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:46.2083631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:46.2085007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:46.2105865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:46.2106872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:47.0422684Z [E ProcessGroupNCCL.cpp:821] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=5, OpType=ALLREDUCE, Timeout(ms)=60000) ran for 66822 milliseconds before timing out.
2022-09-27T16:21:47.2424069Z [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
2022-09-27T16:21:47.2425485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.2426075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.2457703Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-09-27T16:21:47.2458112Z Traceback (most recent call last):
2022-09-27T16:21:47.2459176Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:21:47.2460069Z     getattr(self, test_name)()
2022-09-27T16:21:47.2460674Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:21:47.2461062Z     fn()
2022-09-27T16:21:47.2461553Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T16:21:47.2461942Z     return func(*args, **kwargs)
2022-09-27T16:21:47.2462549Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4816, in test_DistributedDataParallel
2022-09-27T16:21:47.2463034Z     self._test_DistributedDataParallel(
2022-09-27T16:21:47.2463662Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4030, in _test_DistributedDataParallel
2022-09-27T16:21:47.2464093Z     self._test_DDP_niter(
2022-09-27T16:21:47.2464646Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3958, in _test_DDP_niter
2022-09-27T16:21:47.2465068Z     self._assert_equal_param(
2022-09-27T16:21:47.2465618Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3913, in _assert_equal_param
2022-09-27T16:21:47.2466065Z     self.assertEqual(p_gpu, p_DDP)
2022-09-27T16:21:47.2466592Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2435, in assertEqual
2022-09-27T16:21:47.2466977Z     assert_equal(
2022-09-27T16:21:47.2467441Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1093, in assert_equal
2022-09-27T16:21:47.2467835Z     raise error_metas[0].to_error(msg)
2022-09-27T16:21:47.2468212Z AssertionError: Tensor-likes are not close!
2022-09-27T16:21:47.2468419Z 
2022-09-27T16:21:47.2468535Z Mismatched elements: 13 / 20 (65.0%)
2022-09-27T16:21:47.2468994Z Greatest absolute difference: 0.0019322186708450317 at index (5, 0) (up to 1e-05 allowed)
2022-09-27T16:21:47.2469508Z Greatest relative difference: 0.008191716062457405 at index (5, 0) (up to 1.3e-06 allowed)
2022-09-27T16:21:47.2469866Z  exiting process 0 with exit code: 10
2022-09-27T16:21:47.2599351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.2599919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.3878824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.3879374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.4237159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.4237682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.4410459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.4410979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.4644559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.4645125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.4818451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.4819247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.5093132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.5093652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.5366571Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-09-27T16:21:47.5367332Z Traceback (most recent call last):
2022-09-27T16:21:47.5368475Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:21:47.5368894Z     getattr(self, test_name)()
2022-09-27T16:21:47.5369429Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:21:47.5369811Z     fn()
2022-09-27T16:21:47.5370305Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T16:21:47.5370806Z     return func(*args, **kwargs)
2022-09-27T16:21:47.5372085Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4816, in test_DistributedDataParallel
2022-09-27T16:21:47.5372971Z     self._test_DistributedDataParallel(
2022-09-27T16:21:47.5373610Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4030, in _test_DistributedDataParallel
2022-09-27T16:21:47.5374054Z     self._test_DDP_niter(
2022-09-27T16:21:47.5374609Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3958, in _test_DDP_niter
2022-09-27T16:21:47.5375026Z     self._assert_equal_param(
2022-09-27T16:21:47.5375580Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3913, in _assert_equal_param
2022-09-27T16:21:47.5376029Z     self.assertEqual(p_gpu, p_DDP)
2022-09-27T16:21:47.5376560Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2435, in assertEqual
2022-09-27T16:21:47.5376934Z     assert_equal(
2022-09-27T16:21:47.5377395Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1093, in assert_equal
2022-09-27T16:21:47.5377785Z     raise error_metas[0].to_error(msg)
2022-09-27T16:21:47.5378156Z AssertionError: Tensor-likes are not close!
2022-09-27T16:21:47.5378365Z 
2022-09-27T16:21:47.5378481Z Mismatched elements: 13 / 20 (65.0%)
2022-09-27T16:21:47.5378941Z Greatest absolute difference: 0.0019322186708450317 at index (5, 0) (up to 1e-05 allowed)
2022-09-27T16:21:47.5379617Z Greatest relative difference: 0.008191716062457405 at index (5, 0) (up to 1.3e-06 allowed)
2022-09-27T16:21:47.5379992Z  exiting process 1 with exit code: 10
2022-09-27T16:21:47.5496492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.5497460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.5898258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.5899057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.7149993Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-09-27T16:21:47.7161708Z ERROR (74.496s)
2022-09-27T16:21:47.7162482Z     test_DistributedDataParallel errored - num_retries_left: 3
2022-09-27T16:21:47.7163273Z Traceback (most recent call last):
2022-09-27T16:21:47.7164088Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 501, in wrapper
2022-09-27T16:21:47.7164521Z     self._join_processes(fn)
2022-09-27T16:21:47.7165071Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 724, in _join_processes
2022-09-27T16:21:47.7166156Z     self._check_return_codes(elapsed_time)
2022-09-27T16:21:47.7166811Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 769, in _check_return_codes
2022-09-27T16:21:47.7167228Z     raise RuntimeError(error)
2022-09-27T16:21:47.7167561Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-09-27T16:21:47.7167913Z Traceback (most recent call last):
2022-09-27T16:21:47.7168447Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:21:47.7168848Z     getattr(self, test_name)()
2022-09-27T16:21:47.7169355Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:21:47.7169727Z     fn()
2022-09-27T16:21:47.7170222Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 101, in wrapper
2022-09-27T16:21:47.7170641Z     return func(*args, **kwargs)
2022-09-27T16:21:47.7171229Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4816, in test_DistributedDataParallel
2022-09-27T16:21:47.7171711Z     self._test_DistributedDataParallel(
2022-09-27T16:21:47.7172338Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4030, in _test_DistributedDataParallel
2022-09-27T16:21:47.7172759Z     self._test_DDP_niter(
2022-09-27T16:21:47.7173313Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3958, in _test_DDP_niter
2022-09-27T16:21:47.7173733Z     self._assert_equal_param(
2022-09-27T16:21:47.7174542Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 3913, in _assert_equal_param
2022-09-27T16:21:47.7174980Z     self.assertEqual(p_gpu, p_DDP)
2022-09-27T16:21:47.7175517Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2435, in assertEqual
2022-09-27T16:21:47.7175899Z     assert_equal(
2022-09-27T16:21:47.7176361Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1093, in assert_equal
2022-09-27T16:21:47.7176748Z     raise error_metas[0].to_error(msg)
2022-09-27T16:21:47.7177124Z AssertionError: Tensor-likes are not close!
2022-09-27T16:21:47.7177331Z 
2022-09-27T16:21:47.7177463Z Mismatched elements: 13 / 20 (65.0%)
2022-09-27T16:21:47.7177904Z Greatest absolute difference: 0.0019322186708450317 at index (5, 0) (up to 1e-05 allowed)
2022-09-27T16:21:47.7178411Z Greatest relative difference: 0.008191716062457405 at index (5, 0) (up to 1.3e-06 allowed)
2022-09-27T16:21:47.7178645Z 
2022-09-27T16:21:47.7178663Z 
2022-09-27T16:21:47.7178681Z 
2022-09-27T16:21:47.7191874Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66975
2022-09-27T16:21:47.7197834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66976
2022-09-27T16:21:47.7307844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:47.7308631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:48.5245385Z ok (75.301s)
2022-09-27T16:21:48.5245753Z 
2022-09-27T16:21:48.5246161Z ----------------------------------------------------------------------
2022-09-27T16:21:48.5246531Z Ran 1 test in 75.301s
2022-09-27T16:21:48.5246699Z 
2022-09-27T16:21:48.5246799Z OK
2022-09-27T16:21:48.5246917Z 
2022-09-27T16:21:48.5247053Z Generating XML reports...
2022-09-27T16:21:48.5281849Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162033.xml
2022-09-27T16:21:48.8041988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcm6ttk0
2022-09-27T16:21:48.8042608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcm6ttk0/_remote_module_non_scriptable.py
2022-09-27T16:21:48.8799356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcrax7tb
2022-09-27T16:21:48.8800815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcrax7tb/_remote_module_non_scriptable.py
2022-09-27T16:21:49.4181986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:49.4182505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:49.4183087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:49.4183522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:49.4183985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.4184485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.4185136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:49.4185626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:49.4186226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:49.4186701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:49.4371830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.4372351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.4673081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.4673612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.5045273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.5045826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.5233942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.5234494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.5468040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.5468553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.5649003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.5649716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.7009723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.7011450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.8700204Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T16:21:49.8700739Z   warnings.warn(
2022-09-27T16:21:49.8701345Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T16:21:49.8701785Z   warnings.warn(
2022-09-27T16:21:49.8855449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.8855988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:49.9917188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:50.0063633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:50.0064680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:50.0120470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:50.0127192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:50.0286984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.0287545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.1078995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:50.1729310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.1729848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.3115322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.3115881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.3307383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.3307943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.4617875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.4618421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.6015763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.6016325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.6207038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.6207617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.6959405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:50.6959988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:50.6961337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:50.6961835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:50.7463003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.7463791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.7651397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.7651983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.8960769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:50.8961389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:51.0403014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:51.0403573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:51.1844666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:51.1845226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:51.1997211Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:21:51.2020101Z 
2022-09-27T16:21:51.2020351Z Running tests...
2022-09-27T16:21:51.2021071Z ----------------------------------------------------------------------
2022-09-27T16:21:51.3289410Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:51.3290120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:51.3581263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznevtlye
2022-09-27T16:21:51.3581885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznevtlye/_remote_module_non_scriptable.py
2022-09-27T16:21:51.4556031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h1pd0fa
2022-09-27T16:21:51.4557161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h1pd0fa/_remote_module_non_scriptable.py
2022-09-27T16:21:52.0179296Z ok (11.524s)
2022-09-27T16:21:52.0179528Z 
2022-09-27T16:21:52.0180048Z ----------------------------------------------------------------------
2022-09-27T16:21:52.0180607Z Ran 2 tests in 78.833s
2022-09-27T16:21:52.0180783Z 
2022-09-27T16:21:52.0180931Z OK (skipped=1, expected failures=1)
2022-09-27T16:21:52.0181128Z 
2022-09-27T16:21:52.0181255Z Generating XML reports...
2022-09-27T16:21:52.0221148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162033.xml
2022-09-27T16:21:52.8991358Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:21:52.9193298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67097
2022-09-27T16:21:52.9198643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67098
2022-09-27T16:21:53.1804570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.1808251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.2020294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.2045454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.3343814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.3344890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.4690764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.4691830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.4863458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.4864783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.5076848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.5077891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.5252049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.5253081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.6832623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.6851078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.8208384Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T16:21:53.8208890Z   warnings.warn(
2022-09-27T16:21:53.8209493Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1764: UserWarning: You've set static_graph to be True, no need to set it again.
2022-09-27T16:21:53.8209935Z   warnings.warn(
2022-09-27T16:21:53.8353216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.8354006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.9750367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:53.9750901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.1147433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.1148460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.2493675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.2494729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.2660961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.2661966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.2923320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.2924377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.3030094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:54.3031042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:54.3032698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:54.3033632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:54.4282300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.4283357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.4449945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.4450965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.5675282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.5676324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.5841264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.5842261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.6063115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:54.6064223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:54.6065439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:54.6065967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:54.6871838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:54.6872354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:54.6873884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:54.6874372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:54.7112115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.7112670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.8065393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:21:54.8089044Z 
2022-09-27T16:21:54.8089339Z Running tests...
2022-09-27T16:21:54.8089761Z ----------------------------------------------------------------------
2022-09-27T16:21:54.8499565Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.8500204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.9895636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:54.9896165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:55.1002701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:55.1288874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:55.1289873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:55.1892902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:55.1893756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:55.1913444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:55.1920587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:55.2026129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyp4p3_d6
2022-09-27T16:21:55.2029092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyp4p3_d6/_remote_module_non_scriptable.py
2022-09-27T16:21:55.2906413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:55.3016977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0t0452c
2022-09-27T16:21:55.3018434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0t0452c/_remote_module_non_scriptable.py
2022-09-27T16:21:55.8371026Z ok (8.121s)
2022-09-27T16:21:55.8371247Z 
2022-09-27T16:21:55.8371649Z ----------------------------------------------------------------------
2022-09-27T16:21:55.8371991Z Ran 2 tests in 82.617s
2022-09-27T16:21:55.8372162Z 
2022-09-27T16:21:55.8372308Z OK (skipped=1, expected failures=1)
2022-09-27T16:21:55.8372497Z 
2022-09-27T16:21:55.8372626Z Generating XML reports...
2022-09-27T16:21:55.8410371Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162033.xml
2022-09-27T16:21:56.7466944Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:21:56.7669822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67206
2022-09-27T16:21:56.7675770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67207
2022-09-27T16:21:56.8006772Z [1664295716.800118] [28fd6574525b:67097:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:21:56.8007278Z [1664295716.800118] [28fd6574525b:67097:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:21:56.8013122Z [1664295716.800917] [28fd6574525b:67098:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:21:56.8013615Z [1664295716.800917] [28fd6574525b:67098:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:21:57.1627558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:57.1628649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:57.8313960Z ok (6.629s)
2022-09-27T16:21:57.8314352Z 
2022-09-27T16:21:57.8315459Z ----------------------------------------------------------------------
2022-09-27T16:21:57.8316094Z Ran 1 test in 6.629s
2022-09-27T16:21:57.8316437Z 
2022-09-27T16:21:57.8316613Z OK
2022-09-27T16:21:57.8316870Z 
2022-09-27T16:21:57.8317110Z Generating XML reports...
2022-09-27T16:21:57.8350389Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162151.xml
2022-09-27T16:21:58.1005787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:58.1006759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:58.1008321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:58.1009241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:58.4524398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:58.4525380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:58.4526571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:58.4527592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:58.4891265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:58.4892206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:58.4893899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:58.4894885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:21:58.6064848Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:21:58.6089905Z 
2022-09-27T16:21:58.6090374Z Running tests...
2022-09-27T16:21:58.6090898Z ----------------------------------------------------------------------
2022-09-27T16:21:58.6108175Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-09-27T16:21:58.6108652Z 
2022-09-27T16:21:58.6108949Z ----------------------------------------------------------------------
2022-09-27T16:21:58.6109295Z Ran 1 test in 0.002s
2022-09-27T16:21:58.6109463Z 
2022-09-27T16:21:58.6109574Z OK (skipped=1)
2022-09-27T16:21:58.6109716Z 
2022-09-27T16:21:58.6109844Z Generating XML reports...
2022-09-27T16:21:58.6146314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162158.xml
2022-09-27T16:21:59.0614471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:21:59.0615605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:21:59.0616445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:59.0715295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:21:59.0722385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:21:59.0827365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdu29fypv
2022-09-27T16:21:59.0830168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdu29fypv/_remote_module_non_scriptable.py
2022-09-27T16:21:59.1628807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:21:59.1735461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukpsm0rh
2022-09-27T16:21:59.1738199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukpsm0rh/_remote_module_non_scriptable.py
2022-09-27T16:21:59.1958976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:59.1959492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:21:59.5757949Z ok (4.766s)
2022-09-27T16:21:59.5758398Z 
2022-09-27T16:21:59.5759228Z ----------------------------------------------------------------------
2022-09-27T16:21:59.5760029Z Ran 1 test in 4.766s
2022-09-27T16:21:59.5760974Z 
2022-09-27T16:21:59.5761172Z OK
2022-09-27T16:21:59.5761488Z 
2022-09-27T16:21:59.5761795Z Generating XML reports...
2022-09-27T16:21:59.5826156Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162154.xml
2022-09-27T16:21:59.9988059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:21:59.9988564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:21:59.9992834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:21:59.9993341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:00.4968045Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:22:00.4990637Z 
2022-09-27T16:22:00.4991034Z Running tests...
2022-09-27T16:22:00.4991513Z ----------------------------------------------------------------------
2022-09-27T16:22:00.6101398Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:00.6102057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:00.6105049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:00.6105560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:01.1738396Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:22:01.1762011Z 
2022-09-27T16:22:01.1762273Z Running tests...
2022-09-27T16:22:01.1762708Z ----------------------------------------------------------------------
2022-09-27T16:22:01.1780388Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-09-27T16:22:01.1780760Z 
2022-09-27T16:22:01.1781047Z ----------------------------------------------------------------------
2022-09-27T16:22:01.1781380Z Ran 1 test in 0.002s
2022-09-27T16:22:01.1781548Z 
2022-09-27T16:22:01.1781975Z OK (skipped=1)
2022-09-27T16:22:01.1782161Z 
2022-09-27T16:22:01.1782288Z Generating XML reports...
2022-09-27T16:22:01.1816984Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162201.xml
2022-09-27T16:22:01.6895159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:01.6896092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:01.6897194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:01.6898058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:02.0806170Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:02.1000117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67427
2022-09-27T16:22:02.1006575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67428
2022-09-27T16:22:02.2257519Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:22:02.2298966Z 
2022-09-27T16:22:02.2299751Z Running tests...
2022-09-27T16:22:02.2301351Z ----------------------------------------------------------------------
2022-09-27T16:22:03.2522166Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:03.2523435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:03.2524653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:03.2525629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:03.7420651Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:22:03.7442840Z 
2022-09-27T16:22:03.7443308Z Running tests...
2022-09-27T16:22:03.7443801Z ----------------------------------------------------------------------
2022-09-27T16:22:03.7811107Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:03.7812259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:03.7813297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:03.7813774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:03.8464945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:03.8465438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:03.8468409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:03.8468886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:03.8619154Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:03.8822549Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67528
2022-09-27T16:22:03.8828232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67529
2022-09-27T16:22:04.2907485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:04.3705487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:04.3706321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:04.3718138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:04.3725291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:04.3831871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpleh47pbj
2022-09-27T16:22:04.3834263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpleh47pbj/_remote_module_non_scriptable.py
2022-09-27T16:22:04.4718727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:04.4826102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92jusx2i
2022-09-27T16:22:04.4828899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92jusx2i/_remote_module_non_scriptable.py
2022-09-27T16:22:05.4234326Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:05.4436560Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67597
2022-09-27T16:22:05.4441732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67598
2022-09-27T16:22:05.5636869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:05.5637672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:05.5638803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:05.5639296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:05.5815568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:05.5816062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:05.5818299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:05.5818788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:05.9183119Z [1664295725.917807] [28fd6574525b:67427:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:05.9183660Z [1664295725.917807] [28fd6574525b:67427:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:05.9417876Z [1664295725.941377] [28fd6574525b:67428:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:05.9418410Z [1664295725.941377] [28fd6574525b:67428:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:06.2023899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:06.2025276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:06.2027113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:06.2028916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:06.2036931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:06.2096379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:06.2150722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3czkuz79
2022-09-27T16:22:06.2153424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3czkuz79/_remote_module_non_scriptable.py
2022-09-27T16:22:06.2232454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44nvo_n8
2022-09-27T16:22:06.2234705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44nvo_n8/_remote_module_non_scriptable.py
2022-09-27T16:22:06.2469105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:06.2469927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:06.3290681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:06.3291221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:06.6897964Z ok (4.459s)
2022-09-27T16:22:06.6898235Z 
2022-09-27T16:22:06.6898646Z ----------------------------------------------------------------------
2022-09-27T16:22:06.6898988Z Ran 1 test in 4.460s
2022-09-27T16:22:06.6899152Z 
2022-09-27T16:22:06.6899248Z OK
2022-09-27T16:22:06.6899390Z 
2022-09-27T16:22:06.6899509Z Generating XML reports...
2022-09-27T16:22:06.6935323Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162202.xml
2022-09-27T16:22:07.0113972Z ok (6.512s)
2022-09-27T16:22:07.0114204Z 
2022-09-27T16:22:07.0114610Z ----------------------------------------------------------------------
2022-09-27T16:22:07.0115278Z Ran 1 test in 6.512s
2022-09-27T16:22:07.0115572Z 
2022-09-27T16:22:07.0115674Z OK
2022-09-27T16:22:07.0115822Z 
2022-09-27T16:22:07.0115965Z Generating XML reports...
2022-09-27T16:22:07.0150619Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162200.xml
2022-09-27T16:22:07.1582223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:07.1583152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:07.1584287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:07.1585239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:07.1586942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:07.1587843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:07.1593838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:07.1594844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:07.6752883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:07.7045045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:07.7045877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:07.7057211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:07.7064953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:07.8103896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:08.9087428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:08.9088397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:08.9089711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:08.9090253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:09.1425389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbc4j6dl
2022-09-27T16:22:09.1426524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbc4j6dl/_remote_module_non_scriptable.py
2022-09-27T16:22:09.2242408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:09.2243770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:09.2244931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:09.2245905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:09.2392366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhr7bsm9
2022-09-27T16:22:09.2393807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhr7bsm9/_remote_module_non_scriptable.py
2022-09-27T16:22:09.4204753Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:22:09.4228117Z 
2022-09-27T16:22:09.4228340Z Running tests...
2022-09-27T16:22:09.4228821Z ----------------------------------------------------------------------
2022-09-27T16:22:09.7234650Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:22:09.7257663Z 
2022-09-27T16:22:09.7258051Z Running tests...
2022-09-27T16:22:09.7258514Z ----------------------------------------------------------------------
2022-09-27T16:22:10.9363601Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:10.9364280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:10.9567497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:10.9569607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:10.9849991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:10.9850537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:11.0057646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:11.0058214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:11.0735714Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:11.0940572Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67775
2022-09-27T16:22:11.0944683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67776
2022-09-27T16:22:11.1324685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:11.1325695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:11.1783604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:11.1947763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:11.3538353Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:11.3744035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67778
2022-09-27T16:22:11.3748739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67779
2022-09-27T16:22:11.9582264Z ok (8.213s)
2022-09-27T16:22:11.9582492Z 
2022-09-27T16:22:11.9582887Z ----------------------------------------------------------------------
2022-09-27T16:22:11.9583233Z Ran 1 test in 8.214s
2022-09-27T16:22:11.9583403Z 
2022-09-27T16:22:11.9583482Z OK
2022-09-27T16:22:11.9583628Z 
2022-09-27T16:22:11.9583766Z Generating XML reports...
2022-09-27T16:22:11.9618109Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162203.xml
2022-09-27T16:22:12.8004606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:12.8005129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:12.8006377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:12.8006895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:12.8145240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:12.8145724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:12.8149812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:12.8150314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:13.0609592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:13.0610581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:13.0611762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:13.0612710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:13.1408388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:13.1410745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:13.1413828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:13.1414990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:13.4081068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:13.4081667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:13.4082489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:13.4083203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:13.4089438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:13.4089942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:13.6234718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:13.6880737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:13.6881613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:13.6944349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:13.6951824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:13.7893865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:14.1271582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:14.1272492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:14.1273560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:14.1274025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:14.6661655Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:22:14.6686683Z 
2022-09-27T16:22:14.6686992Z Running tests...
2022-09-27T16:22:14.6687430Z ----------------------------------------------------------------------
2022-09-27T16:22:14.8244088Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemwt6qnx
2022-09-27T16:22:14.8245209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemwt6qnx/_remote_module_non_scriptable.py
2022-09-27T16:22:14.8429665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzrs1yga
2022-09-27T16:22:14.8431010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzrs1yga/_remote_module_non_scriptable.py
2022-09-27T16:22:15.0540349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoet3ylbr
2022-09-27T16:22:15.0540978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoet3ylbr/_remote_module_non_scriptable.py
2022-09-27T16:22:15.1562444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mk_lsim
2022-09-27T16:22:15.1563387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mk_lsim/_remote_module_non_scriptable.py
2022-09-27T16:22:15.1925802Z [1664295735.192095] [28fd6574525b:67778:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:15.1926580Z [1664295735.192095] [28fd6574525b:67778:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:15.1937795Z [1664295735.193475] [28fd6574525b:67779:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:15.1938272Z [1664295735.193475] [28fd6574525b:67779:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:15.5465086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.5466110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.5720793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.5721735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.6044218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.6045083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.6291923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.6292949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.7608029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.7608887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.7856772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:15.7857752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.2635576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.2641859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.2872547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.2873432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.3339587Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:16.3537580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67974
2022-09-27T16:22:16.3543229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67975
2022-09-27T16:22:16.4161081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.4162232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.4377283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.4383888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.5061841Z ok (7.083s)
2022-09-27T16:22:16.5062190Z 
2022-09-27T16:22:16.5062973Z ----------------------------------------------------------------------
2022-09-27T16:22:16.5063516Z Ran 1 test in 7.083s
2022-09-27T16:22:16.5063688Z 
2022-09-27T16:22:16.5063787Z OK
2022-09-27T16:22:16.5063926Z 
2022-09-27T16:22:16.5064044Z Generating XML reports...
2022-09-27T16:22:16.5106363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162209.xml
2022-09-27T16:22:16.5654684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.5658958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.5877763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:16.5885261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:17.3892463Z ok (7.663s)
2022-09-27T16:22:17.3892844Z 
2022-09-27T16:22:17.3893520Z ----------------------------------------------------------------------
2022-09-27T16:22:17.3894147Z Ran 1 test in 7.663s
2022-09-27T16:22:17.3894459Z 
2022-09-27T16:22:17.3894632Z OK
2022-09-27T16:22:17.3894885Z 
2022-09-27T16:22:17.3895120Z Generating XML reports...
2022-09-27T16:22:17.3949035Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162209.xml
2022-09-27T16:22:18.0701337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:18.0701904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:18.0702779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:18.0703280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:18.0981798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:18.0982276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:18.0986000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:18.0986504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:18.6068715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:18.6572331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:18.6573094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:18.6577889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:18.6578393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:18.6710312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:18.6711039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:18.6775674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:18.6782314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:18.7726335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:19.1795061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:22:19.1818559Z 
2022-09-27T16:22:19.1818904Z Running tests...
2022-09-27T16:22:19.1819355Z ----------------------------------------------------------------------
2022-09-27T16:22:19.5879345Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:19.5879999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:19.5881378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:19.5881850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:20.0219597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23b47n3p
2022-09-27T16:22:20.0220248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23b47n3p/_remote_module_non_scriptable.py
2022-09-27T16:22:20.1134721Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:22:20.1159364Z 
2022-09-27T16:22:20.1159895Z Running tests...
2022-09-27T16:22:20.1160664Z ----------------------------------------------------------------------
2022-09-27T16:22:20.1525387Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk20j_9a0
2022-09-27T16:22:20.1526443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk20j_9a0/_remote_module_non_scriptable.py
2022-09-27T16:22:20.8216783Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:20.8417670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68120
2022-09-27T16:22:20.8422703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68121
2022-09-27T16:22:21.1837799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:21.1838357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:21.1975011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:21.1975534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:21.7898306Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:21.8096799Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68194
2022-09-27T16:22:21.8103243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68195
2022-09-27T16:22:21.8665457Z ok (7.197s)
2022-09-27T16:22:21.8665798Z 
2022-09-27T16:22:21.8666247Z ----------------------------------------------------------------------
2022-09-27T16:22:21.8666825Z Ran 1 test in 7.198s
2022-09-27T16:22:21.8667004Z 
2022-09-27T16:22:21.8667105Z OK
2022-09-27T16:22:21.8667245Z 
2022-09-27T16:22:21.8667383Z Generating XML reports...
2022-09-27T16:22:21.8702369Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162214.xml
2022-09-27T16:22:22.5251342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:22.5252262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:22.5256391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:22.5257472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:22.5281205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:22.5282230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:22.5289894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:22.5291000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:23.1252719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:23.1253284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:23.1254088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:23.1254802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:23.1260432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:23.1261242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:23.4877059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:23.4877579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:23.4878738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:23.4879520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:23.5686458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:23.5686956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:23.5688235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:23.5688720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:24.0159398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:24.0438866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:24.0439367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:24.0441247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:24.0442014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:24.0996389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:24.0997198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:24.1072785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:24.1079924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:24.2009329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:24.4977789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_4aidy1
2022-09-27T16:22:24.4978899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_4aidy1/_remote_module_non_scriptable.py
2022-09-27T16:22:24.5176175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk75e3gmb
2022-09-27T16:22:24.5177098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk75e3gmb/_remote_module_non_scriptable.py
2022-09-27T16:22:24.5459177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:24.5459732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:24.5548458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:22:24.5572541Z 
2022-09-27T16:22:24.5572848Z Running tests...
2022-09-27T16:22:24.5573535Z ----------------------------------------------------------------------
2022-09-27T16:22:24.5624035Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:24.5625109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:25.1520010Z ok (5.970s)
2022-09-27T16:22:25.1520507Z 
2022-09-27T16:22:25.1520889Z ----------------------------------------------------------------------
2022-09-27T16:22:25.1521239Z Ran 1 test in 5.970s
2022-09-27T16:22:25.1521412Z 
2022-09-27T16:22:25.1521508Z OK
2022-09-27T16:22:25.1521649Z 
2022-09-27T16:22:25.1521791Z Generating XML reports...
2022-09-27T16:22:25.1556552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162219.xml
2022-09-27T16:22:25.5013735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbk6ghwd5
2022-09-27T16:22:25.5014365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbk6ghwd5/_remote_module_non_scriptable.py
2022-09-27T16:22:25.5631802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztdn4248
2022-09-27T16:22:25.5633018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztdn4248/_remote_module_non_scriptable.py
2022-09-27T16:22:25.5911197Z [1664295745.590684] [28fd6574525b:68194:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:25.5911737Z [1664295745.590684] [28fd6574525b:68194:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:25.5984212Z [1664295745.598114] [28fd6574525b:68195:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:25.5984732Z [1664295745.598114] [28fd6574525b:68195:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:26.0025132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:26.0025727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:26.0173036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:26.0173589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:26.2639698Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:26.2843767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68329
2022-09-27T16:22:26.2849596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68330
2022-09-27T16:22:26.6211105Z ok (6.505s)
2022-09-27T16:22:26.6211356Z 
2022-09-27T16:22:26.6211785Z ----------------------------------------------------------------------
2022-09-27T16:22:26.6212138Z Ran 1 test in 6.505s
2022-09-27T16:22:26.6212299Z 
2022-09-27T16:22:26.6212406Z OK
2022-09-27T16:22:26.6212521Z 
2022-09-27T16:22:26.6212661Z Generating XML reports...
2022-09-27T16:22:26.6248665Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162220.xml
2022-09-27T16:22:27.4617293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:27.4617839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:27.4619730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:27.4620220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:27.9665076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:22:27.9677494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:27.9678171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:27.9681914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:27.9687834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:27.9688259Z 
2022-09-27T16:22:27.9688434Z Running tests...
2022-09-27T16:22:27.9688874Z ----------------------------------------------------------------------
2022-09-27T16:22:27.9828543Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:27.9829162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:27.9832598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:27.9833337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:28.4808738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:28.4980063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:28.4981143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:28.5011635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:28.5018852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:28.5996052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:28.7126316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:28.7126870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:28.7128754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:29.2446390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:29.2447085Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:22:29.2471338Z 
2022-09-27T16:22:29.2471510Z Running tests...
2022-09-27T16:22:29.2472374Z ----------------------------------------------------------------------
2022-09-27T16:22:29.6534206Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:29.6744949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68471
2022-09-27T16:22:29.6750041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68472
2022-09-27T16:22:29.8350738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcuz8qkvh
2022-09-27T16:22:29.8351337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcuz8qkvh/_remote_module_non_scriptable.py
2022-09-27T16:22:29.9481709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3hffl9i
2022-09-27T16:22:29.9482326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3hffl9i/_remote_module_non_scriptable.py
2022-09-27T16:22:30.9483357Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:30.9692108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68542
2022-09-27T16:22:30.9698169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68543
2022-09-27T16:22:31.0296716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:31.0298164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:31.0455000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:31.0456484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:31.2834990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:31.2836005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:31.2837198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:31.2838174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:31.3381029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:31.3382031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:31.3383635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:31.3385023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:31.5979593Z ok (7.040s)
2022-09-27T16:22:31.5979866Z 
2022-09-27T16:22:31.5980257Z ----------------------------------------------------------------------
2022-09-27T16:22:31.5980609Z Ran 1 test in 7.041s
2022-09-27T16:22:31.5980779Z 
2022-09-27T16:22:31.5980877Z OK
2022-09-27T16:22:31.5980996Z 
2022-09-27T16:22:31.5983757Z Generating XML reports...
2022-09-27T16:22:31.6019595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162224.xml
2022-09-27T16:22:31.9338487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:31.9339553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:31.9341023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:31.9342394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:31.9448363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:32.0350299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:32.6399628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:32.6400477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:32.6402293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:32.6402827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:32.6736211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:32.6736723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:32.6738929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:32.6739427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:33.1816015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:33.2702856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:33.2703707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:33.2730048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:33.2737553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:33.3499287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ycznne5
2022-09-27T16:22:33.3500364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ycznne5/_remote_module_non_scriptable.py
2022-09-27T16:22:33.3716249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:33.4200801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvb6_pkl
2022-09-27T16:22:33.4201937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvb6_pkl/_remote_module_non_scriptable.py
2022-09-27T16:22:33.4509699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:33.4510259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:33.4702718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:33.4703260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:33.7608666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:33.7609199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:33.7610323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:33.7610798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:33.9849056Z ok (6.016s)
2022-09-27T16:22:33.9849252Z 
2022-09-27T16:22:33.9849705Z ----------------------------------------------------------------------
2022-09-27T16:22:33.9850040Z Ran 1 test in 6.016s
2022-09-27T16:22:33.9850208Z 
2022-09-27T16:22:33.9850324Z OK
2022-09-27T16:22:33.9850460Z 
2022-09-27T16:22:33.9850592Z Generating XML reports...
2022-09-27T16:22:33.9886123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162227.xml
2022-09-27T16:22:34.2707925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:22:34.2730839Z 
2022-09-27T16:22:34.2731293Z Running tests...
2022-09-27T16:22:34.2731808Z ----------------------------------------------------------------------
2022-09-27T16:22:34.6854152Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7ohmz94
2022-09-27T16:22:34.6854930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7ohmz94/_remote_module_non_scriptable.py
2022-09-27T16:22:34.7707765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9sb6e8h
2022-09-27T16:22:34.7708723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9sb6e8h/_remote_module_non_scriptable.py
2022-09-27T16:22:34.7972832Z [1664295754.796823] [28fd6574525b:68542:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:34.7973365Z [1664295754.796823] [28fd6574525b:68542:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:34.8023280Z [1664295754.802002] [28fd6574525b:68543:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:34.8023762Z [1664295754.802002] [28fd6574525b:68543:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:35.2471590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:35.2472194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:35.2701495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:35.2702054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:35.9385164Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:35.9578357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68714
2022-09-27T16:22:35.9584511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68715
2022-09-27T16:22:35.9810820Z ok (6.734s)
2022-09-27T16:22:35.9811003Z 
2022-09-27T16:22:35.9811667Z ----------------------------------------------------------------------
2022-09-27T16:22:35.9812012Z Ran 1 test in 6.734s
2022-09-27T16:22:35.9812182Z 
2022-09-27T16:22:35.9812265Z OK
2022-09-27T16:22:35.9812408Z 
2022-09-27T16:22:35.9812540Z Generating XML reports...
2022-09-27T16:22:35.9867665Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162229.xml
2022-09-27T16:22:36.2037614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:36.2038731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:36.2039952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:36.2040773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:36.8078481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:22:36.8100815Z 
2022-09-27T16:22:36.8100950Z Running tests...
2022-09-27T16:22:36.8101682Z ----------------------------------------------------------------------
2022-09-27T16:22:37.7252000Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:37.7252810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:37.7253421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:37.7253921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:37.7530355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:37.7530836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:37.7533776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:37.7534267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:38.2048189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:38.2048714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:38.2049922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:38.2050424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:38.2440915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:38.2814589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:38.2815442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:38.2846869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:38.2853958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:38.3828946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:38.4424705Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:38.4625634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68824
2022-09-27T16:22:38.4631975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68825
2022-09-27T16:22:38.6995925Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:22:38.7017928Z 
2022-09-27T16:22:38.7018401Z Running tests...
2022-09-27T16:22:38.7018896Z ----------------------------------------------------------------------
2022-09-27T16:22:39.7205699Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah_my6y0
2022-09-27T16:22:39.7206945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah_my6y0/_remote_module_non_scriptable.py
2022-09-27T16:22:39.7886995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1ctp7si
2022-09-27T16:22:39.7888259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1ctp7si/_remote_module_non_scriptable.py
2022-09-27T16:22:40.1595200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:40.1595819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:40.1596932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:40.1597426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:40.1677034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:40.1677493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:40.1680647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:40.1681371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:40.3929550Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:40.4125597Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68895
2022-09-27T16:22:40.4130533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68896
2022-09-27T16:22:40.7323731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:40.7356554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:40.7357364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:40.7425588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:40.7454754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:40.8373523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:40.9498583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:40.9499146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:41.9716391Z ok (7.698s)
2022-09-27T16:22:41.9716638Z 
2022-09-27T16:22:41.9717047Z ----------------------------------------------------------------------
2022-09-27T16:22:41.9717400Z Ran 1 test in 7.698s
2022-09-27T16:22:41.9717553Z 
2022-09-27T16:22:41.9717648Z OK
2022-09-27T16:22:41.9717794Z 
2022-09-27T16:22:41.9717934Z Generating XML reports...
2022-09-27T16:22:41.9754775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162234.xml
2022-09-27T16:22:42.1890103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:42.1891088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:42.1892223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:42.1892800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:42.2626757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:42.2627392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:42.2628988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:42.2629547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:42.3125144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgiwu0561
2022-09-27T16:22:42.3126210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgiwu0561/_remote_module_non_scriptable.py
2022-09-27T16:22:42.4123651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9wdyal3
2022-09-27T16:22:42.4124932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9wdyal3/_remote_module_non_scriptable.py
2022-09-27T16:22:42.4366153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:42.4366758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:42.7282503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:42.7830183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:42.7831069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:42.7892019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:42.7899239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:42.8844482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:43.1750114Z ok (6.364s)
2022-09-27T16:22:43.1750557Z 
2022-09-27T16:22:43.1756872Z ----------------------------------------------------------------------
2022-09-27T16:22:43.1757769Z Ran 1 test in 6.365s
2022-09-27T16:22:43.1758157Z 
2022-09-27T16:22:43.1758378Z OK
2022-09-27T16:22:43.1758672Z 
2022-09-27T16:22:43.1758973Z Generating XML reports...
2022-09-27T16:22:43.1816309Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162236.xml
2022-09-27T16:22:44.2239341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:44.2240775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:44.2242456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:44.2243412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:44.2761103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxqhos74
2022-09-27T16:22:44.2762279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxqhos74/_remote_module_non_scriptable.py
2022-09-27T16:22:44.3292198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wkqbhpz
2022-09-27T16:22:44.3293305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wkqbhpz/_remote_module_non_scriptable.py
2022-09-27T16:22:44.3627226Z [1664295764.362247] [28fd6574525b:68895:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:44.3627769Z [1664295764.362247] [28fd6574525b:68895:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:44.3653049Z [1664295764.365002] [28fd6574525b:68896:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:44.3653547Z [1664295764.365002] [28fd6574525b:68896:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:44.7616413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:44.7616974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:44.7692955Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:22:44.7722443Z 
2022-09-27T16:22:44.7722691Z Running tests...
2022-09-27T16:22:44.7723110Z ----------------------------------------------------------------------
2022-09-27T16:22:44.7750080Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.003s)
2022-09-27T16:22:44.7750809Z 
2022-09-27T16:22:44.7751097Z ----------------------------------------------------------------------
2022-09-27T16:22:44.7751440Z Ran 1 test in 0.003s
2022-09-27T16:22:44.7751585Z 
2022-09-27T16:22:44.7751701Z OK (skipped=1)
2022-09-27T16:22:44.7751861Z 
2022-09-27T16:22:44.7751988Z Generating XML reports...
2022-09-27T16:22:44.7787196Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162244.xml
2022-09-27T16:22:45.4339036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:45.4339566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:45.4342429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:45.4342941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:45.5246271Z ok (6.822s)
2022-09-27T16:22:45.5246482Z 
2022-09-27T16:22:45.5246844Z ----------------------------------------------------------------------
2022-09-27T16:22:45.5247190Z Ran 1 test in 6.823s
2022-09-27T16:22:45.5247359Z 
2022-09-27T16:22:45.5247440Z OK
2022-09-27T16:22:45.5247576Z 
2022-09-27T16:22:45.5247712Z Generating XML reports...
2022-09-27T16:22:45.5281287Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162238.xml
2022-09-27T16:22:45.9873537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:22:45.9896574Z 
2022-09-27T16:22:45.9896856Z Running tests...
2022-09-27T16:22:45.9897320Z ----------------------------------------------------------------------
2022-09-27T16:22:45.9919167Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-09-27T16:22:45.9920140Z 
2022-09-27T16:22:45.9921011Z ----------------------------------------------------------------------
2022-09-27T16:22:45.9921367Z Ran 1 test in 0.002s
2022-09-27T16:22:45.9921538Z 
2022-09-27T16:22:45.9921651Z OK (skipped=1)
2022-09-27T16:22:45.9921811Z 
2022-09-27T16:22:45.9921921Z Generating XML reports...
2022-09-27T16:22:45.9955453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162245.xml
2022-09-27T16:22:46.7601807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:46.7602649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:46.7603960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:46.7604564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:47.3242814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:22:47.3264908Z 
2022-09-27T16:22:47.3265170Z Running tests...
2022-09-27T16:22:47.3265617Z ----------------------------------------------------------------------
2022-09-27T16:22:47.6435811Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:47.6436446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:47.6438865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:47.6439367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:48.0113044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:48.0113860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:48.0114923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:48.0115571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:48.1530000Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:22:48.1553644Z 
2022-09-27T16:22:48.1553925Z Running tests...
2022-09-27T16:22:48.1554619Z ----------------------------------------------------------------------
2022-09-27T16:22:48.1578049Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... skip: test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test (0.002s)
2022-09-27T16:22:48.1578941Z 
2022-09-27T16:22:48.1579504Z ----------------------------------------------------------------------
2022-09-27T16:22:48.1579876Z Ran 1 test in 0.002s
2022-09-27T16:22:48.1580043Z 
2022-09-27T16:22:48.1580155Z OK (skipped=1)
2022-09-27T16:22:48.1580314Z 
2022-09-27T16:22:48.1580424Z Generating XML reports...
2022-09-27T16:22:48.1615449Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162248.xml
2022-09-27T16:22:48.5098369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:22:48.5120962Z 
2022-09-27T16:22:48.5121535Z Running tests...
2022-09-27T16:22:48.5122043Z ----------------------------------------------------------------------
2022-09-27T16:22:48.9087442Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:48.9290438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69170
2022-09-27T16:22:48.9296058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69171
2022-09-27T16:22:50.1798734Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:50.1993348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69266
2022-09-27T16:22:50.1997508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:50.1998243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:50.1998662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69267
2022-09-27T16:22:50.2004530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:50.2005035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:50.6582922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:50.6583483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:50.6584633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:50.6585130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:50.7159916Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:22:50.7183212Z 
2022-09-27T16:22:50.7183508Z Running tests...
2022-09-27T16:22:50.7183966Z ----------------------------------------------------------------------
2022-09-27T16:22:50.7190887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:50.7191357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:50.7194848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:50.7195354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:51.2367942Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:51.3236621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:51.3237440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:51.3279020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:51.3286693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:51.4251983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:51.8394394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:51.8394956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:51.8396129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:51.8396627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:51.8763716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:51.8764255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:51.8765958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:51.8766472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:52.4273484Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:52.4335866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:52.4336728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:52.4337512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:52.4338220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:52.4346330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:52.4346968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:52.4484247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69355
2022-09-27T16:22:52.4491270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69356
2022-09-27T16:22:52.7121431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzr0pqd58
2022-09-27T16:22:52.7122088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzr0pqd58/_remote_module_non_scriptable.py
2022-09-27T16:22:52.8167928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2p40zz9i
2022-09-27T16:22:52.8169442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2p40zz9i/_remote_module_non_scriptable.py
2022-09-27T16:22:53.8770406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1soyevg8
2022-09-27T16:22:53.8771532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1soyevg8/_remote_module_non_scriptable.py
2022-09-27T16:22:53.9159060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxzgngqs
2022-09-27T16:22:53.9160607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxzgngqs/_remote_module_non_scriptable.py
2022-09-27T16:22:54.1089326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:54.1090734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:54.1091907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:54.1092898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:54.1383999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:54.1384996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:54.1386198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:54.1387197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:54.4037535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:54.4038593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:54.4323389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:54.4324009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:54.4324509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:54.4325247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:54.4513929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:54.4515637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:54.6763687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:22:54.6863544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:22:54.6865079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:54.6866454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:22:54.6872558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:22:54.6873501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:22:55.1106752Z ok (6.598s)
2022-09-27T16:22:55.1106960Z 
2022-09-27T16:22:55.1107381Z ----------------------------------------------------------------------
2022-09-27T16:22:55.1107705Z Ran 1 test in 6.598s
2022-09-27T16:22:55.1108131Z 
2022-09-27T16:22:55.1108243Z OK
2022-09-27T16:22:55.1108387Z 
2022-09-27T16:22:55.1108521Z Generating XML reports...
2022-09-27T16:22:55.1144000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162248.xml
2022-09-27T16:22:55.1454511Z ok (7.819s)
2022-09-27T16:22:55.1454716Z 
2022-09-27T16:22:55.1455097Z ----------------------------------------------------------------------
2022-09-27T16:22:55.1455441Z Ran 1 test in 7.819s
2022-09-27T16:22:55.1455613Z 
2022-09-27T16:22:55.1455692Z OK
2022-09-27T16:22:55.1455833Z 
2022-09-27T16:22:55.1455977Z Generating XML reports...
2022-09-27T16:22:55.1493347Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162247.xml
2022-09-27T16:22:56.2247361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp36gdi9pb
2022-09-27T16:22:56.2249140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp36gdi9pb/_remote_module_non_scriptable.py
2022-09-27T16:22:56.2438145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4v0ok_kn
2022-09-27T16:22:56.2440311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4v0ok_kn/_remote_module_non_scriptable.py
2022-09-27T16:22:56.2736833Z [1664295776.272997] [28fd6574525b:69356:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:56.2737904Z [1664295776.272997] [28fd6574525b:69356:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:56.2748629Z [1664295776.274339] [28fd6574525b:69355:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:22:56.2749801Z [1664295776.274339] [28fd6574525b:69355:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:22:57.1322874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:57.1323432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:57.1486857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:57.1487382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:22:57.3922657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:57.3923183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:57.3925356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:57.3925833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:57.3950590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:57.3951091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:57.3955179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:57.3955665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:22:57.7608065Z ok (7.042s)
2022-09-27T16:22:57.7608292Z 
2022-09-27T16:22:57.7608656Z ----------------------------------------------------------------------
2022-09-27T16:22:57.7609003Z Ran 1 test in 7.042s
2022-09-27T16:22:57.7609181Z 
2022-09-27T16:22:57.7609279Z OK
2022-09-27T16:22:57.7609421Z 
2022-09-27T16:22:57.7609559Z Generating XML reports...
2022-09-27T16:22:57.7644489Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162250.xml
2022-09-27T16:22:57.9053801Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:22:57.9076436Z 
2022-09-27T16:22:57.9077024Z Running tests...
2022-09-27T16:22:57.9077483Z ----------------------------------------------------------------------
2022-09-27T16:22:57.9126634Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:22:57.9151180Z 
2022-09-27T16:22:57.9151489Z Running tests...
2022-09-27T16:22:57.9151919Z ----------------------------------------------------------------------
2022-09-27T16:22:59.5701064Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:59.5901718Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69555
2022-09-27T16:22:59.5908468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69556
2022-09-27T16:22:59.6352906Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:22:59.6551276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69558
2022-09-27T16:22:59.6557510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69559
2022-09-27T16:22:59.8014199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:22:59.8014706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:22:59.8017141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:22:59.8017634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:00.4184564Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:00.4221971Z 
2022-09-27T16:23:00.4224330Z Running tests...
2022-09-27T16:23:00.4227093Z ----------------------------------------------------------------------
2022-09-27T16:23:01.3435809Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:01.3436600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:01.3437667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:01.3438143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:01.3464517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:01.3464995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:01.3468542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:01.3469018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:01.3921644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:01.3922159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:01.3924214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:01.3924687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:01.4591688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:01.4592179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:01.4594313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:01.4594786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:01.8654643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:01.8674875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:01.8675742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:01.8755452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:01.8762075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:01.9688498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:02.0163119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:02.0163653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:02.0164465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:02.0165177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:02.0273281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:02.0332878Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:02.0539196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69707
2022-09-27T16:23:02.0545966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69708
2022-09-27T16:23:02.1178649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:03.3077273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx81l9x4a
2022-09-27T16:23:03.3077961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx81l9x4a/_remote_module_non_scriptable.py
2022-09-27T16:23:03.4214603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr2h2pn5
2022-09-27T16:23:03.4215257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr2h2pn5/_remote_module_non_scriptable.py
2022-09-27T16:23:03.4708716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzglkzz5s
2022-09-27T16:23:03.4709677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzglkzz5s/_remote_module_non_scriptable.py
2022-09-27T16:23:03.5245517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3y0f396
2022-09-27T16:23:03.5246420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3y0f396/_remote_module_non_scriptable.py
2022-09-27T16:23:03.5521435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:03.5522028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:03.5751976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:03.5752523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:03.7466685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:03.7467192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:03.7468595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:03.7469546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:03.7528051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:03.7528958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:03.7531704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:03.7532589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:04.1660536Z ok (6.251s)
2022-09-27T16:23:04.1660776Z 
2022-09-27T16:23:04.1661393Z ----------------------------------------------------------------------
2022-09-27T16:23:04.1661732Z Ran 1 test in 6.251s
2022-09-27T16:23:04.1661902Z 
2022-09-27T16:23:04.1662007Z OK
2022-09-27T16:23:04.1662145Z 
2022-09-27T16:23:04.1662287Z Generating XML reports...
2022-09-27T16:23:04.1696641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162257.xml
2022-09-27T16:23:04.2937936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:04.3043513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:04.3044406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:04.3141751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:04.3149014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:04.4058687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:04.5178139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:04.5178700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:04.5314977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:04.5315503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:05.1034880Z ok (7.195s)
2022-09-27T16:23:05.1035135Z 
2022-09-27T16:23:05.1035537Z ----------------------------------------------------------------------
2022-09-27T16:23:05.1035878Z Ran 1 test in 7.196s
2022-09-27T16:23:05.1036062Z 
2022-09-27T16:23:05.1036156Z OK
2022-09-27T16:23:05.1036290Z 
2022-09-27T16:23:05.1036406Z Generating XML reports...
2022-09-27T16:23:05.1073315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162257.xml
2022-09-27T16:23:05.8156475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm7m6l1i
2022-09-27T16:23:05.8157084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm7m6l1i/_remote_module_non_scriptable.py
2022-09-27T16:23:05.8647142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3t7i8kn
2022-09-27T16:23:05.8648281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3t7i8kn/_remote_module_non_scriptable.py
2022-09-27T16:23:05.8911677Z [1664295785.890823] [28fd6574525b:69707:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:23:05.8912188Z [1664295785.890823] [28fd6574525b:69707:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:23:05.9010898Z [1664295785.900795] [28fd6574525b:69708:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:23:05.9011399Z [1664295785.900795] [28fd6574525b:69708:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:23:06.2966487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:06.2971592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:06.3473519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:06.3474752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:06.3869985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:06.3871150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:06.3875755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:06.3876903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:06.9149011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:23:06.9172543Z 
2022-09-27T16:23:06.9173160Z Running tests...
2022-09-27T16:23:06.9173676Z ----------------------------------------------------------------------
2022-09-27T16:23:06.9657335Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... ok (6.544s)
2022-09-27T16:23:06.9657676Z 
2022-09-27T16:23:06.9658056Z ----------------------------------------------------------------------
2022-09-27T16:23:06.9658404Z Ran 1 test in 6.544s
2022-09-27T16:23:06.9658571Z 
2022-09-27T16:23:06.9658671Z OK
2022-09-27T16:23:06.9659067Z 
2022-09-27T16:23:06.9659178Z Generating XML reports...
2022-09-27T16:23:06.9694586Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162300.xml
2022-09-27T16:23:07.3065808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:07.3066311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:07.3067889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:07.3068380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:07.8075372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:23:07.8097317Z 
2022-09-27T16:23:07.8097599Z Running tests...
2022-09-27T16:23:07.8098042Z ----------------------------------------------------------------------
2022-09-27T16:23:08.6092153Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:08.6257547Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.708s)
2022-09-27T16:23:08.6258132Z 
2022-09-27T16:23:08.6258439Z ----------------------------------------------------------------------
2022-09-27T16:23:08.6258780Z Ran 1 test in 1.708s
2022-09-27T16:23:08.6258928Z 
2022-09-27T16:23:08.6259039Z OK (skipped=1)
2022-09-27T16:23:08.6259198Z 
2022-09-27T16:23:08.6259325Z Generating XML reports...
2022-09-27T16:23:08.6292410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162306.xml
2022-09-27T16:23:09.0739301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:09.0740164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:09.0742252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:09.0742755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:09.4282518Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:09.4443289Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-09-27T16:23:09.4444055Z 
2022-09-27T16:23:09.4444357Z ----------------------------------------------------------------------
2022-09-27T16:23:09.4444694Z Ran 1 test in 1.634s
2022-09-27T16:23:09.4444886Z 
2022-09-27T16:23:09.4445001Z OK (skipped=1)
2022-09-27T16:23:09.4445168Z 
2022-09-27T16:23:09.4445299Z Generating XML reports...
2022-09-27T16:23:09.4477393Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162307.xml
2022-09-27T16:23:09.5990249Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:09.6014507Z 
2022-09-27T16:23:09.6014962Z Running tests...
2022-09-27T16:23:09.6015471Z ----------------------------------------------------------------------
2022-09-27T16:23:10.7521586Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:10.7522259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:10.7522872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:10.7523717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:11.2410780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:23:11.2432960Z 
2022-09-27T16:23:11.2433293Z Running tests...
2022-09-27T16:23:11.2433755Z ----------------------------------------------------------------------
2022-09-27T16:23:11.2663962Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:11.2830675Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.681s)
2022-09-27T16:23:11.2831352Z 
2022-09-27T16:23:11.2831632Z ----------------------------------------------------------------------
2022-09-27T16:23:11.2831980Z Ran 1 test in 1.682s
2022-09-27T16:23:11.2832146Z 
2022-09-27T16:23:11.2832255Z OK (skipped=1)
2022-09-27T16:23:11.2832411Z 
2022-09-27T16:23:11.2832521Z Generating XML reports...
2022-09-27T16:23:11.2866805Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162309.xml
2022-09-27T16:23:11.5450328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:11.5450841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:11.5452947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:11.5453454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:12.0424960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:23:12.0447297Z 
2022-09-27T16:23:12.0447557Z Running tests...
2022-09-27T16:23:12.0448001Z ----------------------------------------------------------------------
2022-09-27T16:23:12.8731443Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:12.8930061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70011
2022-09-27T16:23:12.8934996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70012
2022-09-27T16:23:13.4082056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:13.4082554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:13.4085282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:13.4085799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:13.6990239Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:13.7181048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70077
2022-09-27T16:23:13.7187185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70078
2022-09-27T16:23:14.0067333Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:14.0090515Z 
2022-09-27T16:23:14.0090969Z Running tests...
2022-09-27T16:23:14.0091447Z ----------------------------------------------------------------------
2022-09-27T16:23:14.6345509Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:14.6346714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:14.6347905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:14.6349251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:14.7197977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:14.7198810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:14.7199822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:14.7200986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:15.2484841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:15.2485937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:15.2487128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:15.2488547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:15.2493333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:15.2494886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:15.3420460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:15.3421324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:15.3422333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:15.3423140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:15.4067816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:15.4068903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:15.4070158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:15.4071195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:15.6547879Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:15.6743099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70155
2022-09-27T16:23:15.6749893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70156
2022-09-27T16:23:15.7000114Z ok (4.456s)
2022-09-27T16:23:15.7000758Z 
2022-09-27T16:23:15.7001507Z ----------------------------------------------------------------------
2022-09-27T16:23:15.7002404Z Ran 1 test in 4.457s
2022-09-27T16:23:15.7002595Z 
2022-09-27T16:23:15.7002673Z OK
2022-09-27T16:23:15.7002809Z 
2022-09-27T16:23:15.7002948Z Generating XML reports...
2022-09-27T16:23:15.7037848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162311.xml
2022-09-27T16:23:15.8725475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:15.9401590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:15.9402417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:15.9435416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:15.9443222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:16.0415100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:16.5253555Z ok (4.480s)
2022-09-27T16:23:16.5254092Z 
2022-09-27T16:23:16.5254485Z ----------------------------------------------------------------------
2022-09-27T16:23:16.5254837Z Ran 1 test in 4.481s
2022-09-27T16:23:16.5255006Z 
2022-09-27T16:23:16.5255100Z OK
2022-09-27T16:23:16.5255242Z 
2022-09-27T16:23:16.5255379Z Generating XML reports...
2022-09-27T16:23:16.5289683Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162312.xml
2022-09-27T16:23:17.3005003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:17.3005641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:17.3007764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:17.3008263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:17.3663178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:17.3663707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:17.3667006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:17.3667503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:17.8009802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:17.8590827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:17.8591342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:17.8594486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:17.8594986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:17.8696885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:17.8698255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:17.8718401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:17.8725283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:17.9711180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:18.3684398Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:23:18.3707949Z 
2022-09-27T16:23:18.3708566Z Running tests...
2022-09-27T16:23:18.3709038Z ----------------------------------------------------------------------
2022-09-27T16:23:18.3812066Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... ok (4.372s)
2022-09-27T16:23:18.3812616Z 
2022-09-27T16:23:18.3812920Z ----------------------------------------------------------------------
2022-09-27T16:23:18.3813243Z Ran 1 test in 4.372s
2022-09-27T16:23:18.3813409Z 
2022-09-27T16:23:18.3813509Z OK
2022-09-27T16:23:18.3813645Z 
2022-09-27T16:23:18.3813770Z Generating XML reports...
2022-09-27T16:23:18.3849193Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162314.xml
2022-09-27T16:23:18.6251329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:18.6251843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:18.6253061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:18.6253536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:19.1242079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:23:19.1264361Z 
2022-09-27T16:23:19.1264731Z Running tests...
2022-09-27T16:23:19.1265168Z ----------------------------------------------------------------------
2022-09-27T16:23:20.0698886Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:20.0904903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70330
2022-09-27T16:23:20.0910340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70331
2022-09-27T16:23:20.4699260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:20.4700195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:20.4701783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:20.4702746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:20.6711244Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:20.6905979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70334
2022-09-27T16:23:20.6911948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70335
2022-09-27T16:23:20.9631331Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:20.9655838Z 
2022-09-27T16:23:20.9656212Z Running tests...
2022-09-27T16:23:20.9656747Z ----------------------------------------------------------------------
2022-09-27T16:23:21.7964802Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:21.7965458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:21.7966477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:21.7967059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:21.8029136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:21.8029594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:21.8032983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:21.8033471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:22.3413798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:22.3414506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:22.3415307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:22.3416019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:22.3522819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:22.4122182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:22.4122721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:22.4124154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:22.4124643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:22.4308650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:22.4309409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:22.4312075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:22.4312555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:22.4425635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:22.5940874Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:22.6145561Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70476
2022-09-27T16:23:22.6151900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70477
2022-09-27T16:23:22.9429109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:22.9570211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:22.9571058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:22.9631912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:22.9638594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:23.0585286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:23.7596156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnq9_wau6
2022-09-27T16:23:23.7597391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnq9_wau6/_remote_module_non_scriptable.py
2022-09-27T16:23:23.8742544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfyrnhad
2022-09-27T16:23:23.8743738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfyrnhad/_remote_module_non_scriptable.py
2022-09-27T16:23:24.3232017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:24.3233064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:24.3234216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:24.3235150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:24.3278063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:24.3279089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:24.3395039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:24.3396046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:24.3397569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:24.3398118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:24.3542389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsig98tqz
2022-09-27T16:23:24.3544453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsig98tqz/_remote_module_non_scriptable.py
2022-09-27T16:23:24.3921628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:24.3922196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:24.4331965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kn2e066
2022-09-27T16:23:24.4332589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kn2e066/_remote_module_non_scriptable.py
2022-09-27T16:23:24.8767164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:24.8776136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:24.8776957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:24.8870453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:24.8877879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:24.9793055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:25.0307904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:25.0308901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:25.0875584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:25.0876572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:25.1016983Z ok (6.731s)
2022-09-27T16:23:25.1017322Z 
2022-09-27T16:23:25.1018000Z ----------------------------------------------------------------------
2022-09-27T16:23:25.1018613Z Ran 1 test in 6.731s
2022-09-27T16:23:25.1018935Z 
2022-09-27T16:23:25.1019120Z OK
2022-09-27T16:23:25.1019377Z 
2022-09-27T16:23:25.1019617Z Generating XML reports...
2022-09-27T16:23:25.1056461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162318.xml
2022-09-27T16:23:25.7019852Z ok (6.575s)
2022-09-27T16:23:25.7020086Z 
2022-09-27T16:23:25.7020501Z ----------------------------------------------------------------------
2022-09-27T16:23:25.7020849Z Ran 1 test in 6.575s
2022-09-27T16:23:25.7021003Z 
2022-09-27T16:23:25.7021109Z OK
2022-09-27T16:23:25.7021246Z 
2022-09-27T16:23:25.7021383Z Generating XML reports...
2022-09-27T16:23:25.7057901Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162319.xml
2022-09-27T16:23:26.4141253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpp21kv8
2022-09-27T16:23:26.4142499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpp21kv8/_remote_module_non_scriptable.py
2022-09-27T16:23:26.5036083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3587x8_
2022-09-27T16:23:26.5037610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3587x8_/_remote_module_non_scriptable.py
2022-09-27T16:23:26.5300242Z [1664295806.529515] [28fd6574525b:70476:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:23:26.5301449Z [1664295806.529515] [28fd6574525b:70476:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:23:26.5331291Z [1664295806.532809] [28fd6574525b:70477:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:23:26.5331796Z [1664295806.532809] [28fd6574525b:70477:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:23:27.3865182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:27.3865747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:27.4408062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:27.4408585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:27.4498675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:27.4499476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:27.4502336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:27.4502816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:27.9441807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:27.9442370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:27.9444682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:27.9445187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:27.9574470Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:23:27.9598816Z 
2022-09-27T16:23:27.9599077Z Running tests...
2022-09-27T16:23:27.9599527Z ----------------------------------------------------------------------
2022-09-27T16:23:28.0282584Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... ok (7.062s)
2022-09-27T16:23:28.0282893Z 
2022-09-27T16:23:28.0283259Z ----------------------------------------------------------------------
2022-09-27T16:23:28.0283599Z Ran 1 test in 7.063s
2022-09-27T16:23:28.0283763Z 
2022-09-27T16:23:28.0283860Z OK
2022-09-27T16:23:28.0283997Z 
2022-09-27T16:23:28.0284108Z Generating XML reports...
2022-09-27T16:23:28.0318579Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162320.xml
2022-09-27T16:23:28.4704602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:23:28.4729090Z 
2022-09-27T16:23:28.4729562Z Running tests...
2022-09-27T16:23:28.4730213Z ----------------------------------------------------------------------
2022-09-27T16:23:29.6552543Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:29.6748533Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70683
2022-09-27T16:23:29.6754117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70684
2022-09-27T16:23:30.0787450Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:30.0986503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70686
2022-09-27T16:23:30.0992760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70687
2022-09-27T16:23:30.1380558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:30.1381037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:30.1383831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:30.1384359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:30.6952122Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:30.6976272Z 
2022-09-27T16:23:30.6976615Z Running tests...
2022-09-27T16:23:30.6977067Z ----------------------------------------------------------------------
2022-09-27T16:23:31.3482644Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:31.3483270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:31.3484417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:31.3484914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:31.3855945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:31.3856797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:31.3858662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:31.3859158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:31.7930130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:31.7930653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:31.7933047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:31.7933547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:31.8098937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:31.8099432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:31.8102476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:31.8102946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:31.9082794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:31.9083349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:31.9084076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:31.9084781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:31.9192646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:32.0098144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:32.3156551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:32.3394252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:32.3395095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:32.3461652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:32.3468843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:32.3726450Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:32.3935218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70835
2022-09-27T16:23:32.3940933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70836
2022-09-27T16:23:32.4407748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:33.9849335Z ok (6.025s)
2022-09-27T16:23:33.9849582Z 
2022-09-27T16:23:33.9849984Z ----------------------------------------------------------------------
2022-09-27T16:23:33.9850335Z Ran 1 test in 6.025s
2022-09-27T16:23:33.9850509Z 
2022-09-27T16:23:33.9850609Z OK
2022-09-27T16:23:33.9850727Z 
2022-09-27T16:23:33.9850866Z Generating XML reports...
2022-09-27T16:23:33.9885885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162327.xml
2022-09-27T16:23:34.0938979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:34.0939490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:34.0940909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:34.0941749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:34.1017084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:34.1017576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:34.1020346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:34.1020840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:34.6411819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:34.6425172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:34.6425997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:34.6513264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:34.6521010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:34.7440682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:35.6116212Z ok (7.138s)
2022-09-27T16:23:35.6116658Z 
2022-09-27T16:23:35.6117339Z ----------------------------------------------------------------------
2022-09-27T16:23:35.6117984Z Ran 1 test in 7.139s
2022-09-27T16:23:35.6118309Z 
2022-09-27T16:23:35.6118511Z OK
2022-09-27T16:23:35.6118805Z 
2022-09-27T16:23:35.6119084Z Generating XML reports...
2022-09-27T16:23:35.6283852Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162328.xml
2022-09-27T16:23:36.1285305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:36.1286282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:36.1287689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:36.1288644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:36.1938615Z [1664295816.193257] [28fd6574525b:70835:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:23:36.1939810Z [1664295816.193257] [28fd6574525b:70835:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:23:36.2019803Z [1664295816.201632] [28fd6574525b:70836:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:23:36.2020334Z [1664295816.201632] [28fd6574525b:70836:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:23:36.6425399Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:23:36.6447817Z 
2022-09-27T16:23:36.6448609Z Running tests...
2022-09-27T16:23:36.6449136Z ----------------------------------------------------------------------
2022-09-27T16:23:36.6468439Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-09-27T16:23:36.6468930Z 
2022-09-27T16:23:36.6469340Z ----------------------------------------------------------------------
2022-09-27T16:23:36.6469663Z Ran 1 test in 0.002s
2022-09-27T16:23:36.6469831Z 
2022-09-27T16:23:36.6469944Z OK (skipped=1)
2022-09-27T16:23:36.6470105Z 
2022-09-27T16:23:36.6470240Z Generating XML reports...
2022-09-27T16:23:36.6504871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162336.xml
2022-09-27T16:23:37.2048589Z ok (6.507s)
2022-09-27T16:23:37.2049268Z 
2022-09-27T16:23:37.2049666Z ----------------------------------------------------------------------
2022-09-27T16:23:37.2050005Z Ran 1 test in 6.507s
2022-09-27T16:23:37.2050158Z 
2022-09-27T16:23:37.2050255Z OK
2022-09-27T16:23:37.2050392Z 
2022-09-27T16:23:37.2050526Z Generating XML reports...
2022-09-27T16:23:37.2085232Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162330.xml
2022-09-27T16:23:37.8557513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:37.8558034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:37.8559502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:37.8559981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:38.3476970Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:23:38.3501756Z 
2022-09-27T16:23:38.3502570Z Running tests...
2022-09-27T16:23:38.3503526Z ----------------------------------------------------------------------
2022-09-27T16:23:38.3531526Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.003s)
2022-09-27T16:23:38.3532202Z 
2022-09-27T16:23:38.3533225Z ----------------------------------------------------------------------
2022-09-27T16:23:38.3534279Z Ran 1 test in 0.003s
2022-09-27T16:23:38.3534613Z 
2022-09-27T16:23:38.3534831Z OK (skipped=1)
2022-09-27T16:23:38.3535546Z 
2022-09-27T16:23:38.3535798Z Generating XML reports...
2022-09-27T16:23:38.3578248Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162338.xml
2022-09-27T16:23:38.6778076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:38.6778620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:38.6780871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:38.6781354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:39.1760430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:23:39.1784038Z 
2022-09-27T16:23:39.1784390Z Running tests...
2022-09-27T16:23:39.1784842Z ----------------------------------------------------------------------
2022-09-27T16:23:39.1801725Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:39.2570916Z Runs _test_accumulate_gradients_no_sync using default inputs ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:39.2572049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:39.2573714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:39.2574652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:39.7614301Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:39.7637701Z 
2022-09-27T16:23:39.7638124Z Running tests...
2022-09-27T16:23:39.7639128Z ----------------------------------------------------------------------
2022-09-27T16:23:39.7659792Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-09-27T16:23:39.7660412Z 
2022-09-27T16:23:39.7660958Z ----------------------------------------------------------------------
2022-09-27T16:23:39.7661614Z Ran 1 test in 0.002s
2022-09-27T16:23:39.7661912Z 
2022-09-27T16:23:39.7662114Z OK (skipped=1)
2022-09-27T16:23:39.7662404Z 
2022-09-27T16:23:39.7662643Z Generating XML reports...
2022-09-27T16:23:39.7698354Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162339.xml
2022-09-27T16:23:40.2146984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:40.2147929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:40.2149109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:40.2150065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:40.7127948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:23:40.7152842Z 
2022-09-27T16:23:40.7153375Z Running tests...
2022-09-27T16:23:40.7154307Z ----------------------------------------------------------------------
2022-09-27T16:23:40.7174140Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:40.7512274Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:40.7882996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71124
2022-09-27T16:23:40.7934949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71125
2022-09-27T16:23:41.7501273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:41.7501780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:41.7504640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:41.7505158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:42.2600684Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:42.2624226Z 
2022-09-27T16:23:42.2624537Z Running tests...
2022-09-27T16:23:42.2625010Z ----------------------------------------------------------------------
2022-09-27T16:23:42.2643074Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:42.2644492Z Runs _test_accumulate_gradients_no_sync using default inputs ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T16:23:42.2644807Z 
2022-09-27T16:23:42.2645114Z ----------------------------------------------------------------------
2022-09-27T16:23:42.2645451Z Ran 1 test in 0.002s
2022-09-27T16:23:42.2645616Z 
2022-09-27T16:23:42.2645709Z OK (skipped=1)
2022-09-27T16:23:42.2645865Z 
2022-09-27T16:23:42.2645991Z Generating XML reports...
2022-09-27T16:23:42.2681254Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162342.xml
2022-09-27T16:23:42.3563391Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:42.3762270Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71190
2022-09-27T16:23:42.3767906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71191
2022-09-27T16:23:42.4515029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:42.4515541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:42.4516943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:42.4517439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:42.5422237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:42.5422943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:42.5426544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:42.5427493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:43.0834984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:43.0845072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:43.0848159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:43.0939770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:43.1002801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:43.1180789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfugbrv7
2022-09-27T16:23:43.1184572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfugbrv7/_remote_module_non_scriptable.py
2022-09-27T16:23:43.1957752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:43.2067592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnc8jpamk
2022-09-27T16:23:43.2070505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnc8jpamk/_remote_module_non_scriptable.py
2022-09-27T16:23:43.7007977Z ok (4.522s)
2022-09-27T16:23:43.7008377Z 
2022-09-27T16:23:43.7009051Z ----------------------------------------------------------------------
2022-09-27T16:23:43.7009701Z Ran 1 test in 4.522s
2022-09-27T16:23:43.7010011Z 
2022-09-27T16:23:43.7010161Z OK
2022-09-27T16:23:43.7010409Z 
2022-09-27T16:23:43.7010652Z Generating XML reports...
2022-09-27T16:23:43.7045864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162339.xml
2022-09-27T16:23:44.0665370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:44.0666386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:44.0668039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:44.0668940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:44.0869829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:44.0870794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:44.0872849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:44.0874126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:44.3220664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:44.3221664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:44.3223308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:44.3224223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:44.5748668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:44.6048620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:44.6050009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:44.6052554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:44.6060517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:44.7063782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:44.8121100Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:44.8147514Z 
2022-09-27T16:23:44.8148210Z Running tests...
2022-09-27T16:23:44.8149062Z ----------------------------------------------------------------------
2022-09-27T16:23:44.8177137Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:44.8178566Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T16:23:44.8179231Z 
2022-09-27T16:23:44.8179794Z ----------------------------------------------------------------------
2022-09-27T16:23:44.8180445Z Ran 1 test in 0.003s
2022-09-27T16:23:44.8180775Z 
2022-09-27T16:23:44.8181461Z OK (skipped=1)
2022-09-27T16:23:44.8181862Z 
2022-09-27T16:23:44.8184982Z Generating XML reports...
2022-09-27T16:23:44.8218961Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162344.xml
2022-09-27T16:23:45.7587065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:45.7587608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:45.7589761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:45.7590264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:45.9422415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzx03x2na
2022-09-27T16:23:45.9423342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzx03x2na/_remote_module_non_scriptable.py
2022-09-27T16:23:46.0889088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbip88j_
2022-09-27T16:23:46.0890274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbip88j_/_remote_module_non_scriptable.py
2022-09-27T16:23:46.2600827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:23:46.2626448Z 
2022-09-27T16:23:46.2626915Z Running tests...
2022-09-27T16:23:46.2627406Z ----------------------------------------------------------------------
2022-09-27T16:23:46.2648519Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:46.7555061Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:46.7556208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:46.7557849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:46.7558880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:47.2658581Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:47.2682115Z 
2022-09-27T16:23:47.2682540Z Running tests...
2022-09-27T16:23:47.2683375Z ----------------------------------------------------------------------
2022-09-27T16:23:47.2706381Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:47.2707558Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T16:23:47.2708262Z 
2022-09-27T16:23:47.2708828Z ----------------------------------------------------------------------
2022-09-27T16:23:47.2709466Z Ran 1 test in 0.003s
2022-09-27T16:23:47.2709821Z 
2022-09-27T16:23:47.2710044Z OK (skipped=1)
2022-09-27T16:23:47.2710359Z 
2022-09-27T16:23:47.2710616Z Generating XML reports...
2022-09-27T16:23:47.2744635Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162347.xml
2022-09-27T16:23:47.8667978Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:47.8870125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71392
2022-09-27T16:23:47.8875369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71393
2022-09-27T16:23:48.3897721Z ok (7.674s)
2022-09-27T16:23:48.3898103Z 
2022-09-27T16:23:48.3898794Z ----------------------------------------------------------------------
2022-09-27T16:23:48.3899422Z Ran 1 test in 7.674s
2022-09-27T16:23:48.3899732Z 
2022-09-27T16:23:48.3899899Z OK
2022-09-27T16:23:48.3900152Z 
2022-09-27T16:23:48.3900394Z Generating XML reports...
2022-09-27T16:23:48.4006186Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162340.xml
2022-09-27T16:23:49.3394349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:49.3394906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:49.3396443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:49.3396948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:49.5985534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:49.5986044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:49.5987071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:49.5987579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:49.7024496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:49.7025026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:49.7026028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:49.7026509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:49.8441954Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:49.8464993Z 
2022-09-27T16:23:49.8465736Z Running tests...
2022-09-27T16:23:49.8466320Z ----------------------------------------------------------------------
2022-09-27T16:23:49.8482583Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:49.8483649Z Runs _test_accumulate_gradients_no_sync using default inputs ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T16:23:49.8484233Z 
2022-09-27T16:23:49.8484851Z ----------------------------------------------------------------------
2022-09-27T16:23:49.8485538Z Ran 1 test in 0.002s
2022-09-27T16:23:49.8485719Z 
2022-09-27T16:23:49.8485812Z OK (skipped=1)
2022-09-27T16:23:49.8485976Z 
2022-09-27T16:23:49.8486105Z Generating XML reports...
2022-09-27T16:23:49.8519300Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162349.xml
2022-09-27T16:23:50.2271736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:50.2272816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:50.2273637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:50.2274352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:50.2379921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:50.2493698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphk728zew
2022-09-27T16:23:50.2496567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphk728zew/_remote_module_non_scriptable.py
2022-09-27T16:23:50.3285538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:50.3397006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd351hc59
2022-09-27T16:23:50.3398639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd351hc59/_remote_module_non_scriptable.py
2022-09-27T16:23:50.3672350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:50.3673367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:50.4593916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:50.4594817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:50.4595880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:50.4596688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:50.7958565Z ok (4.533s)
2022-09-27T16:23:50.7963631Z 
2022-09-27T16:23:50.7966370Z ----------------------------------------------------------------------
2022-09-27T16:23:50.7969114Z Ran 1 test in 4.533s
2022-09-27T16:23:50.7971418Z 
2022-09-27T16:23:50.7973504Z OK
2022-09-27T16:23:50.7977500Z 
2022-09-27T16:23:50.7984262Z Generating XML reports...
2022-09-27T16:23:50.8058835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162346.xml
2022-09-27T16:23:51.0214276Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:23:51.0236243Z 
2022-09-27T16:23:51.0237028Z Running tests...
2022-09-27T16:23:51.0237543Z ----------------------------------------------------------------------
2022-09-27T16:23:51.0258162Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:51.7882501Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:51.7883066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:51.7885335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:51.7885828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:52.3516985Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:23:52.3540199Z 
2022-09-27T16:23:52.3540613Z Running tests...
2022-09-27T16:23:52.3541137Z ----------------------------------------------------------------------
2022-09-27T16:23:52.6632857Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:52.6831332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71603
2022-09-27T16:23:52.6836855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71604
2022-09-27T16:23:52.8748388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:52.8748876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:52.8751511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:52.8752018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:53.4296342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:23:53.4336261Z 
2022-09-27T16:23:53.4338140Z Running tests...
2022-09-27T16:23:53.4339945Z ----------------------------------------------------------------------
2022-09-27T16:23:53.4364686Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:23:53.9955452Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:54.0159087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71670
2022-09-27T16:23:54.0166124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71671
2022-09-27T16:23:54.3288279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:54.3289037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:54.3291580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:54.3292092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:54.4015357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:54.4015920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:54.4016882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:54.4017355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:54.9369124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:54.9892792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:54.9893691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:54.9978203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:54.9985068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:55.0462050Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:23:55.0658676Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71743
2022-09-27T16:23:55.0665408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71744
2022-09-27T16:23:55.0905374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:55.6579386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:55.6579918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:55.6582209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:55.6582725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:55.7421434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:55.7428232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:55.7432880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:55.7436720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:56.2237348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:56.3059695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:56.3060544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:56.3150132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:56.3156911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:56.3224824Z STAGE:2022-09-27 16:23:56 71670:71670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:23:56.3549491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvifpsem2
2022-09-27T16:23:56.3550941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvifpsem2/_remote_module_non_scriptable.py
2022-09-27T16:23:56.4074770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:56.4176180Z STAGE:2022-09-27 16:23:56 71671:71671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:23:56.4925948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc53ai2p6
2022-09-27T16:23:56.4926845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc53ai2p6/_remote_module_non_scriptable.py
2022-09-27T16:23:56.7392611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:56.7393118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:56.7394405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:56.7394895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:56.7526241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:23:56.7526746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:23:56.7528587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:23:56.7529089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:23:57.3279139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:23:57.3279685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:23:57.3280821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:57.3281535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:23:57.3288617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:23:57.3289529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:23:57.3407016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_tej3qk
2022-09-27T16:23:57.3407612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrvu3mvy
2022-09-27T16:23:57.3408172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_tej3qk/_remote_module_non_scriptable.py
2022-09-27T16:23:57.3408715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrvu3mvy/_remote_module_non_scriptable.py
2022-09-27T16:23:57.3695001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:57.3695543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:57.8735696Z ok (4.440s)
2022-09-27T16:23:57.8735933Z 
2022-09-27T16:23:57.8736532Z ----------------------------------------------------------------------
2022-09-27T16:23:57.8736903Z Ran 1 test in 4.440s
2022-09-27T16:23:57.8737053Z 
2022-09-27T16:23:57.8737160Z OK
2022-09-27T16:23:57.8737298Z 
2022-09-27T16:23:57.8737434Z Generating XML reports...
2022-09-27T16:23:57.8773564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162353.xml
2022-09-27T16:23:58.0522285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:58.0522830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:23:58.1030534Z [1664295838.102570] [28fd6574525b:71670:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:23:58.1031051Z [1664295838.102570] [28fd6574525b:71670:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:23:58.1385835Z [1664295838.138123] [28fd6574525b:71671:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:23:58.1386352Z [1664295838.138123] [28fd6574525b:71671:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:23:58.5374251Z STAGE:2022-09-27 16:23:58 71670:71670 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:23:58.5374868Z STAGE:2022-09-27 16:23:58 71671:71671 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:23:58.6639399Z STAGE:2022-09-27 16:23:58 71670:71670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:23:58.6676756Z STAGE:2022-09-27 16:23:58 71671:71671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:23:58.6686742Z STAGE:2022-09-27 16:23:58 71670:71670 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:23:58.6687345Z STAGE:2022-09-27 16:23:58 71671:71671 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:23:58.6966815Z ok (7.673s)
2022-09-27T16:23:58.6967062Z 
2022-09-27T16:23:58.6967637Z ----------------------------------------------------------------------
2022-09-27T16:23:58.6968014Z Ran 1 test in 7.673s
2022-09-27T16:23:58.6968203Z 
2022-09-27T16:23:58.6968280Z OK
2022-09-27T16:23:58.6968418Z 
2022-09-27T16:23:58.6968554Z Generating XML reports...
2022-09-27T16:23:58.7004291Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162351.xml
2022-09-27T16:23:59.8290714Z ok (7.475s)
2022-09-27T16:23:59.8290945Z 
2022-09-27T16:23:59.8291510Z ----------------------------------------------------------------------
2022-09-27T16:23:59.8291864Z Ran 1 test in 7.475s
2022-09-27T16:23:59.8292034Z 
2022-09-27T16:23:59.8292131Z OK
2022-09-27T16:23:59.8292249Z 
2022-09-27T16:23:59.8292386Z Generating XML reports...
2022-09-27T16:23:59.8327195Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162352.xml
2022-09-27T16:24:00.1377614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:00.1378149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:00.1379389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:00.1379881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:00.6265508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:00.6287770Z 
2022-09-27T16:24:00.6288070Z Running tests...
2022-09-27T16:24:00.6288508Z ----------------------------------------------------------------------
2022-09-27T16:24:00.6305725Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:24:00.9289400Z Runs _test_accumulate_gradients_no_sync using default inputs ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:00.9291275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:00.9296336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:00.9298405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:01.4574132Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:01.4597566Z 
2022-09-27T16:24:01.4597816Z Running tests...
2022-09-27T16:24:01.4598267Z ----------------------------------------------------------------------
2022-09-27T16:24:01.4620325Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-09-27T16:24:01.8694413Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:01.8695012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:01.8695912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:01.8696414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:02.2743055Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:02.2944508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71954
2022-09-27T16:24:02.2949570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71955
2022-09-27T16:24:02.3795532Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:02.3819280Z 
2022-09-27T16:24:02.3819506Z Running tests...
2022-09-27T16:24:02.3819933Z ----------------------------------------------------------------------
2022-09-27T16:24:02.3839878Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-09-27T16:24:02.3840462Z 
2022-09-27T16:24:02.3841141Z ----------------------------------------------------------------------
2022-09-27T16:24:02.3841531Z Ran 1 test in 0.002s
2022-09-27T16:24:02.3841701Z 
2022-09-27T16:24:02.3841826Z OK (skipped=1)
2022-09-27T16:24:02.3841984Z 
2022-09-27T16:24:02.3842115Z Generating XML reports...
2022-09-27T16:24:02.3876439Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162402.xml
2022-09-27T16:24:03.1410678Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:03.1611480Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72020
2022-09-27T16:24:03.1617554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72021
2022-09-27T16:24:04.0746845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:04.0747827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:04.0748965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:04.0749450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:04.1038409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:04.1039148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:04.1041207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:04.1041808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:04.5235403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:04.5235950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:04.5238011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:04.5238843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:04.6376020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:04.6376589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:04.6377362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:04.6378072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:04.6384864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:04.6385387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:04.6500117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwibmv8yh
2022-09-27T16:24:04.6502092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwibmv8yh/_remote_module_non_scriptable.py
2022-09-27T16:24:04.6512289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2std0nto
2022-09-27T16:24:04.6515714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2std0nto/_remote_module_non_scriptable.py
2022-09-27T16:24:04.9135400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:04.9136271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:04.9137358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:04.9138221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:04.9789874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:04.9791116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:04.9792246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:04.9793144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:05.0411637Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:05.0434920Z 
2022-09-27T16:24:05.0435337Z Running tests...
2022-09-27T16:24:05.0436050Z ----------------------------------------------------------------------
2022-09-27T16:24:05.0454971Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-09-27T16:24:05.0455541Z 
2022-09-27T16:24:05.0456108Z ----------------------------------------------------------------------
2022-09-27T16:24:05.0456656Z Ran 1 test in 0.002s
2022-09-27T16:24:05.0456869Z 
2022-09-27T16:24:05.0456995Z OK (skipped=1)
2022-09-27T16:24:05.0457155Z 
2022-09-27T16:24:05.0457265Z Generating XML reports...
2022-09-27T16:24:05.0493125Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162405.xml
2022-09-27T16:24:05.1020340Z ok (4.473s)
2022-09-27T16:24:05.1020550Z 
2022-09-27T16:24:05.1021038Z ----------------------------------------------------------------------
2022-09-27T16:24:05.1021503Z Ran 1 test in 4.473s
2022-09-27T16:24:05.1021687Z 
2022-09-27T16:24:05.1021784Z OK
2022-09-27T16:24:05.1021924Z 
2022-09-27T16:24:05.1022061Z Generating XML reports...
2022-09-27T16:24:05.1058924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162400.xml
2022-09-27T16:24:05.4375443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:05.5031257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:05.5032380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:05.5085428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:05.5092229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:05.6043416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:06.9150454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyh6h6c_
2022-09-27T16:24:06.9151381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyh6h6c_/_remote_module_non_scriptable.py
2022-09-27T16:24:07.0035574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hjroqup
2022-09-27T16:24:07.0036487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hjroqup/_remote_module_non_scriptable.py
2022-09-27T16:24:07.1524683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:07.1525175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:07.1527600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:07.1528096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:07.3170191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:07.3170667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:07.3172014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:07.3172502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:07.6637784Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:07.6662561Z 
2022-09-27T16:24:07.6662863Z Running tests...
2022-09-27T16:24:07.6663308Z ----------------------------------------------------------------------
2022-09-27T16:24:07.6682212Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-09-27T16:24:07.6682578Z 
2022-09-27T16:24:07.6682907Z ----------------------------------------------------------------------
2022-09-27T16:24:07.6683265Z Ran 1 test in 0.002s
2022-09-27T16:24:07.6683411Z 
2022-09-27T16:24:07.6683525Z OK (skipped=1)
2022-09-27T16:24:07.6683688Z 
2022-09-27T16:24:07.6683818Z Generating XML reports...
2022-09-27T16:24:07.6719173Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162407.xml
2022-09-27T16:24:07.8188200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:07.8210738Z 
2022-09-27T16:24:07.8211059Z Running tests...
2022-09-27T16:24:07.8211503Z ----------------------------------------------------------------------
2022-09-27T16:24:08.5566943Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:24:08.5575139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:24:09.1752106Z ok (7.715s)
2022-09-27T16:24:09.1752442Z 
2022-09-27T16:24:09.1753035Z ----------------------------------------------------------------------
2022-09-27T16:24:09.1753385Z Ran 1 test in 7.715s
2022-09-27T16:24:09.1753553Z 
2022-09-27T16:24:09.1753631Z OK
2022-09-27T16:24:09.1753771Z 
2022-09-27T16:24:09.1754606Z Generating XML reports...
2022-09-27T16:24:09.1789329Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162401.xml
2022-09-27T16:24:09.5474052Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:09.5670511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72252
2022-09-27T16:24:09.5676113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72253
2022-09-27T16:24:09.7095896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:09.7096392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:09.7097675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:09.7098225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:10.2690626Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:10.2742124Z 
2022-09-27T16:24:10.2742409Z Running tests...
2022-09-27T16:24:10.2743312Z ----------------------------------------------------------------------
2022-09-27T16:24:10.2784260Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.003s)
2022-09-27T16:24:10.2792262Z 
2022-09-27T16:24:10.2793589Z ----------------------------------------------------------------------
2022-09-27T16:24:10.2795110Z Ran 1 test in 0.005s
2022-09-27T16:24:10.2795419Z 
2022-09-27T16:24:10.2803733Z OK (skipped=1)
2022-09-27T16:24:10.2804191Z 
2022-09-27T16:24:10.2804479Z Generating XML reports...
2022-09-27T16:24:10.2894316Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162410.xml
2022-09-27T16:24:11.2171381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:11.2171896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:11.2174145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:11.2174659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:11.2278470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:11.2278939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:11.2283371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:11.2283862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:11.4477199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:11.4477988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:11.4479415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:11.4479901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:11.7946688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:11.7947257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:11.7948037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:11.7948741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:11.8054246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:11.8118522Z STAGE:2022-09-27 16:24:11 72253:72253 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:11.8958082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:11.9022919Z STAGE:2022-09-27 16:24:11 72252:72252 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:11.9038883Z STAGE:2022-09-27 16:24:11 72252:72252 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:11 72253:72253 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:11.9039292Z 
2022-09-27T16:24:11.9066503Z STAGE:2022-09-27 16:24:11 72252:72252 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:24:11 72253:72253 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:11.9066889Z 
2022-09-27T16:24:11.9079815Z STAGE:2022-09-27 16:24:11 72253:72253 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:11.9080607Z STAGE:2022-09-27 16:24:11 72252:72252 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:11.9481978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:11.9507310Z 
2022-09-27T16:24:11.9507782Z Running tests...
2022-09-27T16:24:11.9508380Z ----------------------------------------------------------------------
2022-09-27T16:24:11.9526551Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-09-27T16:24:12.2747544Z Runs _test_accumulate_gradients_no_sync using default inputs ... ok (4.453s)
2022-09-27T16:24:12.2747856Z 
2022-09-27T16:24:12.2748228Z ----------------------------------------------------------------------
2022-09-27T16:24:12.2748552Z Ran 1 test in 4.454s
2022-09-27T16:24:12.2748719Z 
2022-09-27T16:24:12.2748812Z OK
2022-09-27T16:24:12.2748951Z 
2022-09-27T16:24:12.2749078Z Generating XML reports...
2022-09-27T16:24:12.2784265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162407.xml
2022-09-27T16:24:12.3168343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:12.3168887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:12.3170871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:12.3171362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:12.8225368Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:12.8247772Z 
2022-09-27T16:24:12.8248197Z Running tests...
2022-09-27T16:24:12.8248723Z ----------------------------------------------------------------------
2022-09-27T16:24:12.8277388Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.003s)
2022-09-27T16:24:12.8278023Z 
2022-09-27T16:24:12.8278337Z ----------------------------------------------------------------------
2022-09-27T16:24:12.8278657Z Ran 1 test in 0.003s
2022-09-27T16:24:12.8278834Z 
2022-09-27T16:24:12.8278945Z OK (skipped=1)
2022-09-27T16:24:12.8279101Z 
2022-09-27T16:24:12.8279233Z Generating XML reports...
2022-09-27T16:24:12.8312843Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162412.xml
2022-09-27T16:24:13.6435828Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:13.6634347Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72432
2022-09-27T16:24:13.6640802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72433
2022-09-27T16:24:14.5736937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:14.5737525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:14.5738654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:14.5739182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:14.8866186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:14.8866953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:14.8867951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:14.8868455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:15.0645584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:15.0669428Z 
2022-09-27T16:24:15.0669673Z Running tests...
2022-09-27T16:24:15.0670145Z ----------------------------------------------------------------------
2022-09-27T16:24:15.3350579Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:15.3351319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:15.3352628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:15.3353130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:15.3768959Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:15.3791696Z 
2022-09-27T16:24:15.3792163Z Running tests...
2022-09-27T16:24:15.3792661Z ----------------------------------------------------------------------
2022-09-27T16:24:15.4070042Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:15.4071021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:15.4072161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:15.4072664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:15.8541616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:15.9470185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:15.9471030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:15.9556506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:15.9564286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:16.0485512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:16.6812837Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:16.7023846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72542
2022-09-27T16:24:16.7029183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72543
2022-09-27T16:24:17.0246824Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:17.0449200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72545
2022-09-27T16:24:17.0454581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72546
2022-09-27T16:24:17.2946740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0_i8uiv
2022-09-27T16:24:17.2947580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0_i8uiv/_remote_module_non_scriptable.py
2022-09-27T16:24:17.4093185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpik1__gn2
2022-09-27T16:24:17.4094540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpik1__gn2/_remote_module_non_scriptable.py
2022-09-27T16:24:18.4018566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:18.4019320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:18.4020575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:18.4021102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:18.4438719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:18.4439381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:18.4440639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:18.4441149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:18.7672480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:18.7673006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:18.7674588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:18.7675061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:18.7861841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:18.7862359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:18.7865503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:18.7865999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:18.9839195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:18.9840653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:18.9841480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:18.9842179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:18.9847950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:18.9848906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:18.9950362Z STAGE:2022-09-27 16:24:18 72542:72542 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:18.9951120Z STAGE:2022-09-27 16:24:18 72543:72543 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:18.9968132Z STAGE:2022-09-27 16:24:18 72543:72543 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:18 72542:72542 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:18.9968611Z 
2022-09-27T16:24:18.9986792Z STAGE:2022-09-27 16:24:18 72543:72543 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:18.9987746Z STAGE:2022-09-27 16:24:18 72542:72542 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:19.0001739Z STAGE:2022-09-27 16:24:18 72543:72543 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:19.0002699Z STAGE:2022-09-27 16:24:18 72542:72542 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:19.0020066Z STAGE:2022-09-27 16:24:19 72543:72543 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:19.0021029Z STAGE:2022-09-27 16:24:19 72542:72542 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:19.0034311Z STAGE:2022-09-27 16:24:19 72543:72543 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:19.0035273Z STAGE:2022-09-27 16:24:19 72542:72542 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:19.3032471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:19.3288249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:19.3289331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:19.3339359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:19.3345548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:19.3416681Z STAGE:2022-09-27 16:24:19 72545:72545 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:19.4302367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:19.4367507Z STAGE:2022-09-27 16:24:19 72546:72546 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:19.5097739Z ok (4.442s)
2022-09-27T16:24:19.5097960Z 
2022-09-27T16:24:19.5098573Z ----------------------------------------------------------------------
2022-09-27T16:24:19.5098955Z Ran 1 test in 4.443s
2022-09-27T16:24:19.5099128Z 
2022-09-27T16:24:19.5099233Z OK
2022-09-27T16:24:19.5099371Z 
2022-09-27T16:24:19.5099487Z Generating XML reports...
2022-09-27T16:24:19.5135468Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162415.xml
2022-09-27T16:24:19.6774914Z ok (7.726s)
2022-09-27T16:24:19.6775137Z 
2022-09-27T16:24:19.6775515Z ----------------------------------------------------------------------
2022-09-27T16:24:19.6775840Z Ran 1 test in 7.727s
2022-09-27T16:24:19.6776029Z 
2022-09-27T16:24:19.6776125Z OK
2022-09-27T16:24:19.6776261Z 
2022-09-27T16:24:19.6776394Z Generating XML reports...
2022-09-27T16:24:19.6813173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162411.xml
2022-09-27T16:24:21.2701328Z [1664295861.269622] [28fd6574525b:72545:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:24:21.2701866Z [1664295861.269622] [28fd6574525b:72545:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:24:21.3417767Z [1664295861.341313] [28fd6574525b:72546:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:24:21.3418544Z [1664295861.341313] [28fd6574525b:72546:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:24:21.7238667Z STAGE:2022-09-27 16:24:21 72546:72546 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:21 72545:72545 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:21.7239139Z 
2022-09-27T16:24:21.7239891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:21.7240663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:21.7244548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:21.7245049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:21.9056091Z STAGE:2022-09-27 16:24:21 72546:72546 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:21.9308708Z STAGE:2022-09-27 16:24:21 72545:72545 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:21.9319522Z STAGE:2022-09-27 16:24:21 72546:72546 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:21 72545:72545 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:21.9320430Z 
2022-09-27T16:24:21.9613543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:21.9614027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:21.9617397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:21.9617893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:22.2349550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:22.2373787Z 
2022-09-27T16:24:22.2374228Z Running tests...
2022-09-27T16:24:22.2374691Z ----------------------------------------------------------------------
2022-09-27T16:24:22.4795235Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:22.4818186Z 
2022-09-27T16:24:22.4818334Z Running tests...
2022-09-27T16:24:22.4819028Z ----------------------------------------------------------------------
2022-09-27T16:24:22.4836978Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:24:22.4837643Z 
2022-09-27T16:24:22.4837950Z ----------------------------------------------------------------------
2022-09-27T16:24:22.4838293Z Ran 1 test in 0.002s
2022-09-27T16:24:22.4838458Z 
2022-09-27T16:24:22.4838578Z OK (skipped=1)
2022-09-27T16:24:22.4838741Z 
2022-09-27T16:24:22.4838870Z Generating XML reports...
2022-09-27T16:24:22.4873145Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162422.xml
2022-09-27T16:24:23.0584680Z ok (7.679s)
2022-09-27T16:24:23.0584894Z 
2022-09-27T16:24:23.0585303Z ----------------------------------------------------------------------
2022-09-27T16:24:23.0585644Z Ran 1 test in 7.679s
2022-09-27T16:24:23.0585789Z 
2022-09-27T16:24:23.0585899Z OK
2022-09-27T16:24:23.0586035Z 
2022-09-27T16:24:23.0586173Z Generating XML reports...
2022-09-27T16:24:23.0622900Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162415.xml
2022-09-27T16:24:24.0363017Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:24.0916664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72815
2022-09-27T16:24:24.0922853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72816
2022-09-27T16:24:24.4866918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:24.4867442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:24.4869214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:24.4869728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:25.0467755Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:25.0490405Z 
2022-09-27T16:24:25.0490550Z Running tests...
2022-09-27T16:24:25.0491344Z ----------------------------------------------------------------------
2022-09-27T16:24:25.0508614Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-09-27T16:24:25.0508975Z 
2022-09-27T16:24:25.0509267Z ----------------------------------------------------------------------
2022-09-27T16:24:25.0509629Z Ran 1 test in 0.002s
2022-09-27T16:24:25.0509798Z 
2022-09-27T16:24:25.0509916Z OK (skipped=1)
2022-09-27T16:24:25.0510078Z 
2022-09-27T16:24:25.0510189Z Generating XML reports...
2022-09-27T16:24:25.0544543Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162425.xml
2022-09-27T16:24:25.2417931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:25.2418434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:25.2420449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:25.2420927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:25.7242775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:25.7243782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:25.7244735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:25.7245246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:25.7441131Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:25.7466118Z 
2022-09-27T16:24:25.7466480Z Running tests...
2022-09-27T16:24:25.7466978Z ----------------------------------------------------------------------
2022-09-27T16:24:25.7486235Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T16:24:25.7486562Z 
2022-09-27T16:24:25.7486854Z ----------------------------------------------------------------------
2022-09-27T16:24:25.7487170Z Ran 1 test in 0.002s
2022-09-27T16:24:25.7487335Z 
2022-09-27T16:24:25.7487447Z OK (skipped=1)
2022-09-27T16:24:25.7487606Z 
2022-09-27T16:24:25.7487746Z Generating XML reports...
2022-09-27T16:24:25.7521825Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162425.xml
2022-09-27T16:24:25.7576129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:25.7576607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:25.7580205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:25.7580705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:26.3090866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:26.3091479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:26.3092524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:26.3093270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:26.3199238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:26.4103093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:26.4212821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:24:26.4213349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:24:26.4214100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:24:26.4214783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:24:26.4309622Z STAGE:2022-09-27 16:24:26 72815:72815 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:26.4310776Z STAGE:2022-09-27 16:24:26 72816:72816 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:26.4325435Z STAGE:2022-09-27 16:24:26 72816:72816 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:26.4326030Z STAGE:2022-09-27 16:24:26 72815:72815 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:26.4341587Z STAGE:2022-09-27 16:24:26 72816:72816 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:26.4342171Z STAGE:2022-09-27 16:24:26 72815:72815 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:26.4355351Z STAGE:2022-09-27 16:24:26 72815:72815 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:26 72816:72816 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:26.4355765Z 
2022-09-27T16:24:26.4370314Z STAGE:2022-09-27 16:24:26 72815:72815 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:26.4371286Z STAGE:2022-09-27 16:24:26 72816:72816 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:26.4382301Z STAGE:2022-09-27 16:24:26 72816:72816 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:26.4382900Z STAGE:2022-09-27 16:24:26 72815:72815 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:26.8990706Z ok (4.661s)
2022-09-27T16:24:26.8990900Z 
2022-09-27T16:24:26.8991997Z ----------------------------------------------------------------------
2022-09-27T16:24:26.8992397Z Ran 1 test in 4.662s
2022-09-27T16:24:26.8992571Z 
2022-09-27T16:24:26.8992666Z OK
2022-09-27T16:24:26.8992787Z 
2022-09-27T16:24:26.8992924Z Generating XML reports...
2022-09-27T16:24:26.9026589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162422.xml
2022-09-27T16:24:26.9818829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:26.9819355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:26.9821960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:26.9822460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:27.4754870Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:27.4777795Z 
2022-09-27T16:24:27.4778189Z Running tests...
2022-09-27T16:24:27.4778926Z ----------------------------------------------------------------------
2022-09-27T16:24:27.4796374Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-09-27T16:24:27.4796970Z 
2022-09-27T16:24:27.4797608Z ----------------------------------------------------------------------
2022-09-27T16:24:27.4798132Z Ran 1 test in 0.002s
2022-09-27T16:24:27.4798296Z 
2022-09-27T16:24:27.4798393Z OK (skipped=1)
2022-09-27T16:24:27.4798558Z 
2022-09-27T16:24:27.4798696Z Generating XML reports...
2022-09-27T16:24:27.4832111Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162427.xml
2022-09-27T16:24:27.7794659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:27.7795156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:27.7798391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:27.7799140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:28.2826369Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:28.2851502Z 
2022-09-27T16:24:28.2851939Z Running tests...
2022-09-27T16:24:28.2852826Z ----------------------------------------------------------------------
2022-09-27T16:24:28.2873530Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T16:24:28.2874184Z 
2022-09-27T16:24:28.2874744Z ----------------------------------------------------------------------
2022-09-27T16:24:28.2875367Z Ran 1 test in 0.002s
2022-09-27T16:24:28.2875690Z 
2022-09-27T16:24:28.2875891Z OK (skipped=1)
2022-09-27T16:24:28.2876170Z 
2022-09-27T16:24:28.2876411Z Generating XML reports...
2022-09-27T16:24:28.2912571Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162428.xml
2022-09-27T16:24:29.0632344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:29.0632869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:29.0634360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:29.0634866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:29.4746227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:29.4746722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:29.4748504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:29.4749001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:29.6073071Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:29.6096576Z 
2022-09-27T16:24:29.6096723Z Running tests...
2022-09-27T16:24:29.6097187Z ----------------------------------------------------------------------
2022-09-27T16:24:29.9687828Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:29.9709666Z 
2022-09-27T16:24:29.9709949Z Running tests...
2022-09-27T16:24:29.9710390Z ----------------------------------------------------------------------
2022-09-27T16:24:29.9728808Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-09-27T16:24:29.9729155Z 
2022-09-27T16:24:29.9729428Z ----------------------------------------------------------------------
2022-09-27T16:24:29.9729761Z Ran 1 test in 0.002s
2022-09-27T16:24:29.9729931Z 
2022-09-27T16:24:29.9730040Z OK (skipped=1)
2022-09-27T16:24:29.9730200Z 
2022-09-27T16:24:29.9730308Z Generating XML reports...
2022-09-27T16:24:29.9764890Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162429.xml
2022-09-27T16:24:30.2923366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:30.2924100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:30.2925755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:30.2926230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:30.7999685Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:30.8023327Z 
2022-09-27T16:24:30.8023769Z Running tests...
2022-09-27T16:24:30.8024271Z ----------------------------------------------------------------------
2022-09-27T16:24:31.2525295Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:31.2726338Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73132
2022-09-27T16:24:31.2732282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73133
2022-09-27T16:24:31.9082022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:31.9082537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:31.9084673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:31.9085167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:32.4798683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:32.4813777Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:32.4822026Z 
2022-09-27T16:24:32.4822293Z Running tests...
2022-09-27T16:24:32.4822761Z ----------------------------------------------------------------------
2022-09-27T16:24:32.4840773Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-09-27T16:24:32.4841392Z 
2022-09-27T16:24:32.4841684Z ----------------------------------------------------------------------
2022-09-27T16:24:32.4842016Z Ran 1 test in 0.002s
2022-09-27T16:24:32.4842165Z 
2022-09-27T16:24:32.4842281Z OK (skipped=1)
2022-09-27T16:24:32.4842439Z 
2022-09-27T16:24:32.4842567Z Generating XML reports...
2022-09-27T16:24:32.4877916Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162432.xml
2022-09-27T16:24:32.5033686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73198
2022-09-27T16:24:32.5039916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73199
2022-09-27T16:24:32.9540552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:32.9541497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:32.9542638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:32.9543536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:33.0010373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:33.0011382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:33.0012559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:33.0013513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:33.5912597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:33.5913380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:33.5914184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:33.5914907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:33.6020349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:33.6926014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:34.0798994Z skip: Skipped due to small world size. (4.470s)
2022-09-27T16:24:34.0799353Z 
2022-09-27T16:24:34.0799743Z ----------------------------------------------------------------------
2022-09-27T16:24:34.0800083Z Ran 1 test in 4.470s
2022-09-27T16:24:34.0800458Z 
2022-09-27T16:24:34.0800568Z OK (skipped=1)
2022-09-27T16:24:34.0800712Z 
2022-09-27T16:24:34.0800861Z Generating XML reports...
2022-09-27T16:24:34.0837060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162429.xml
2022-09-27T16:24:34.1920872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:34.1921376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:34.1922615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:34.1923105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:34.2166271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:34.2166764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:34.2169345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:34.2169835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:34.5026984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:34.5027492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:34.5029948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:34.5030449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:34.7049064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:34.7386985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:34.7387833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:34.7453825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:34.7460588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:34.7463655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:24:34.8398409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:34.8402278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:24:34.8403739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:24:34.8477081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:24:34.8478940Z STAGE:2022-09-27 16:24:34 73199:73199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:34.8534983Z STAGE:2022-09-27 16:24:34 73198:73198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:34.9973110Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:34.9995738Z 
2022-09-27T16:24:34.9995954Z Running tests...
2022-09-27T16:24:34.9996405Z ----------------------------------------------------------------------
2022-09-27T16:24:35.0024357Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-09-27T16:24:35.0024904Z 
2022-09-27T16:24:35.0025202Z ----------------------------------------------------------------------
2022-09-27T16:24:35.0025539Z Ran 1 test in 0.003s
2022-09-27T16:24:35.0025707Z 
2022-09-27T16:24:35.0025820Z OK (skipped=1)
2022-09-27T16:24:35.0025977Z 
2022-09-27T16:24:35.0026085Z Generating XML reports...
2022-09-27T16:24:35.0061288Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162434.xml
2022-09-27T16:24:36.2359951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:36.2361286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:36.2362174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:36.2362653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:36.5873113Z [1664295876.586816] [28fd6574525b:73198:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:24:36.5873622Z [1664295876.586816] [28fd6574525b:73198:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:24:36.6207206Z [1664295876.620272] [28fd6574525b:73199:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:24:36.6207716Z [1664295876.620272] [28fd6574525b:73199:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:24:36.7384086Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:36.7407332Z 
2022-09-27T16:24:36.7407479Z Running tests...
2022-09-27T16:24:36.7408421Z ----------------------------------------------------------------------
2022-09-27T16:24:37.0106422Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:24:37 73198:73198 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:37 73199:73199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:37.0106978Z 
2022-09-27T16:24:37.0393084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:37.0393628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:37.0395716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:37.0396572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:37.1377400Z STAGE:2022-09-27 16:24:37 73199:73199 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:37.1445728Z STAGE:2022-09-27 16:24:37 73198:73198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:37.1456888Z STAGE:2022-09-27 16:24:37 73198:73198 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:37.1457961Z STAGE:2022-09-27 16:24:37 73199:73199 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:37.5452318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:37.5474368Z 
2022-09-27T16:24:37.5475021Z Running tests...
2022-09-27T16:24:37.5476012Z ----------------------------------------------------------------------
2022-09-27T16:24:37.5493619Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:24:37.5494325Z 
2022-09-27T16:24:37.5494912Z ----------------------------------------------------------------------
2022-09-27T16:24:37.5495325Z Ran 1 test in 0.002s
2022-09-27T16:24:37.5495495Z 
2022-09-27T16:24:37.5495605Z OK (skipped=1)
2022-09-27T16:24:37.5495762Z 
2022-09-27T16:24:37.5495888Z Generating XML reports...
2022-09-27T16:24:37.5528916Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162437.xml
2022-09-27T16:24:38.2908634Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:38.3101959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73391
2022-09-27T16:24:38.3108173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73392
2022-09-27T16:24:38.3163287Z ok (7.514s)
2022-09-27T16:24:38.3163654Z 
2022-09-27T16:24:38.3164170Z ----------------------------------------------------------------------
2022-09-27T16:24:38.3164798Z Ran 1 test in 7.514s
2022-09-27T16:24:38.3164968Z 
2022-09-27T16:24:38.3165065Z OK
2022-09-27T16:24:38.3165193Z 
2022-09-27T16:24:38.3165326Z Generating XML reports...
2022-09-27T16:24:38.3201985Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162430.xml
2022-09-27T16:24:39.6135483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:39.6136037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:39.6138281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:39.6138778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:40.0030592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:40.0031091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:40.0033311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:40.0033787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:40.0379616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:40.0380100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:40.0382398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:40.0382869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:40.1232797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:40.1257738Z 
2022-09-27T16:24:40.1258117Z Running tests...
2022-09-27T16:24:40.1258557Z ----------------------------------------------------------------------
2022-09-27T16:24:40.4991094Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:40.4991708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:40.4994567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:40.4995068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:40.5700000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:40.5700745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:40.5701552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:40.5702268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:40.5708917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:40.5709395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:40.5811939Z STAGE:2022-09-27 16:24:40 73391:73391 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:40.5812549Z STAGE:2022-09-27 16:24:40 73392:73392 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:40.5826999Z STAGE:2022-09-27 16:24:40 73391:73391 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:40 73392:73392 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:40.5827413Z 
2022-09-27T16:24:40.5844133Z STAGE:2022-09-27 16:24:40 73391:73391 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:24:40 73392:73392 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:40.5844801Z 
2022-09-27T16:24:40.5857534Z STAGE:2022-09-27 16:24:40 73392:73392 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:40 73391:73391 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:40.5857946Z 
2022-09-27T16:24:40.5871281Z STAGE:2022-09-27 16:24:40 73392:73392 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:24:40 73391:73391 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:40.5871678Z 
2022-09-27T16:24:40.5883263Z STAGE:2022-09-27 16:24:40 73391:73391 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:40.5883875Z STAGE:2022-09-27 16:24:40 73392:73392 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:41.0071478Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:41.0094510Z 
2022-09-27T16:24:41.0095023Z Running tests...
2022-09-27T16:24:41.0095508Z ----------------------------------------------------------------------
2022-09-27T16:24:41.0173784Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... ok (4.276s)
2022-09-27T16:24:41.0174241Z 
2022-09-27T16:24:41.0174535Z ----------------------------------------------------------------------
2022-09-27T16:24:41.0174874Z Ran 1 test in 4.277s
2022-09-27T16:24:41.0175048Z 
2022-09-27T16:24:41.0175127Z OK
2022-09-27T16:24:41.0175261Z 
2022-09-27T16:24:41.0175420Z Generating XML reports...
2022-09-27T16:24:41.0212938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162436.xml
2022-09-27T16:24:41.7865607Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:41.8070907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73538
2022-09-27T16:24:41.8077193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73539
2022-09-27T16:24:42.7580138Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:42.7777992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73634
2022-09-27T16:24:42.7784120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73635
2022-09-27T16:24:43.2969142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:43.2969660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:43.2971239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:43.2972257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:43.5776773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:43.5777768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:43.5778989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:43.5779935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:43.6337836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:43.6338396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:43.6340324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:43.6341090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:43.7989332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:43.8012184Z 
2022-09-27T16:24:43.8012393Z Running tests...
2022-09-27T16:24:43.8013041Z ----------------------------------------------------------------------
2022-09-27T16:24:44.0973431Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:44.1498143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:44.1499000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:44.1580923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:44.1588206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:44.2512443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:44.4392191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:44.4392723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:44.4395131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:44.4395889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:44.4426752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:44.4427401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:44.4431015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:44.4431786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:44.9864678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:44.9966878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:44.9967723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:44.9968427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:44.9975576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:44.9976940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:45.3860459Z skip: Skipped due to small world size. (4.376s)
2022-09-27T16:24:45.3860759Z 
2022-09-27T16:24:45.3861410Z ----------------------------------------------------------------------
2022-09-27T16:24:45.3861766Z Ran 1 test in 4.377s
2022-09-27T16:24:45.3861931Z 
2022-09-27T16:24:45.3862041Z OK (skipped=1)
2022-09-27T16:24:45.3862181Z 
2022-09-27T16:24:45.3862320Z Generating XML reports...
2022-09-27T16:24:45.3917986Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162441.xml
2022-09-27T16:24:45.4669451Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:45.4959876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73716
2022-09-27T16:24:45.4965482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73717
2022-09-27T16:24:45.7652208Z STAGE:2022-09-27 16:24:45 73539:73539 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:45.8366784Z STAGE:2022-09-27 16:24:45 73538:73538 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:47.2365438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:47.2366006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:47.2367290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:47.2368100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:47.2535553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:47.2536037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:47.2543102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:47.2543593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:47.3103856Z STAGE:2022-09-27 16:24:47 73538:73538 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:47.3106399Z STAGE:2022-09-27 16:24:47 73539:73539 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:47.4867902Z STAGE:2022-09-27 16:24:47 73538:73538 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:47.4884707Z STAGE:2022-09-27 16:24:47 73538:73538 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:47.4947087Z STAGE:2022-09-27 16:24:47 73539:73539 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:47.4965288Z STAGE:2022-09-27 16:24:47 73539:73539 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:47.6911968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:47.6912941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:47.6914121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:47.6915056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:47.8110775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:47.8111389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:47.8112194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:47.8112885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:47.8119583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:47.8120078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:47.8223510Z STAGE:2022-09-27 16:24:47 73717:73717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:47.8224475Z STAGE:2022-09-27 16:24:47 73716:73716 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:47.8239933Z STAGE:2022-09-27 16:24:47 73717:73717 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:24:47 73716:73716 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:47.8240723Z 
2022-09-27T16:24:48.2012201Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:48.2029152Z ok (4.402s)
2022-09-27T16:24:48.2029337Z 
2022-09-27T16:24:48.2029871Z ----------------------------------------------------------------------
2022-09-27T16:24:48.2030245Z Ran 1 test in 4.402s
2022-09-27T16:24:48.2030413Z 
2022-09-27T16:24:48.2030510Z OK
2022-09-27T16:24:48.2030645Z 
2022-09-27T16:24:48.2030774Z Generating XML reports...
2022-09-27T16:24:48.2033583Z 
2022-09-27T16:24:48.2033952Z Running tests...
2022-09-27T16:24:48.2034469Z ----------------------------------------------------------------------
2022-09-27T16:24:48.2054181Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T16:24:48.2054970Z 
2022-09-27T16:24:48.2055274Z ----------------------------------------------------------------------
2022-09-27T16:24:48.2055603Z Ran 1 test in 0.002s
2022-09-27T16:24:48.2055768Z 
2022-09-27T16:24:48.2055876Z OK (skipped=1)
2022-09-27T16:24:48.2056015Z 
2022-09-27T16:24:48.2056140Z Generating XML reports...
2022-09-27T16:24:48.2067624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162443.xml
2022-09-27T16:24:48.2089035Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162448.xml
2022-09-27T16:24:49.1243680Z ok (8.998s)
2022-09-27T16:24:49.1243934Z 
2022-09-27T16:24:49.1244336Z ----------------------------------------------------------------------
2022-09-27T16:24:49.1244694Z Ran 1 test in 8.998s
2022-09-27T16:24:49.1244844Z 
2022-09-27T16:24:49.1244940Z OK
2022-09-27T16:24:49.1245074Z 
2022-09-27T16:24:49.1245206Z Generating XML reports...
2022-09-27T16:24:49.1281755Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162440.xml
2022-09-27T16:24:50.2436797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:50.2437311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:50.2439345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:50.2439822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:50.2990977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:50.2991462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:50.2994962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:50.2995444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:50.7499362Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:50.7520898Z 
2022-09-27T16:24:50.7521151Z Running tests...
2022-09-27T16:24:50.7521581Z ----------------------------------------------------------------------
2022-09-27T16:24:50.7541499Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T16:24:50.7541868Z 
2022-09-27T16:24:50.7542196Z ----------------------------------------------------------------------
2022-09-27T16:24:50.7542531Z Ran 1 test in 0.002s
2022-09-27T16:24:50.7542684Z 
2022-09-27T16:24:50.7542794Z OK (skipped=1)
2022-09-27T16:24:50.7543239Z 
2022-09-27T16:24:50.7543381Z Generating XML reports...
2022-09-27T16:24:50.7576987Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162450.xml
2022-09-27T16:24:50.7825496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:50.7848875Z 
2022-09-27T16:24:50.7849136Z Running tests...
2022-09-27T16:24:50.7849591Z ----------------------------------------------------------------------
2022-09-27T16:24:51.3148525Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:51.3149116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:51.3151173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:51.3151686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:51.8248767Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:24:51.8272099Z 
2022-09-27T16:24:51.8272407Z Running tests...
2022-09-27T16:24:51.8272882Z ----------------------------------------------------------------------
2022-09-27T16:24:52.3683416Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:52.3880691Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73973
2022-09-27T16:24:52.3886944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73974
2022-09-27T16:24:52.7105137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:52.7105669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:52.7107989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:52.7108469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:53.2619352Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:53.2641716Z 
2022-09-27T16:24:53.2641917Z Running tests...
2022-09-27T16:24:53.2642352Z ----------------------------------------------------------------------
2022-09-27T16:24:53.2660745Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:24:53.2661275Z 
2022-09-27T16:24:53.2661578Z ----------------------------------------------------------------------
2022-09-27T16:24:53.2661929Z Ran 1 test in 0.002s
2022-09-27T16:24:53.2662096Z 
2022-09-27T16:24:53.2662195Z OK (skipped=1)
2022-09-27T16:24:53.2662350Z 
2022-09-27T16:24:53.2662480Z Generating XML reports...
2022-09-27T16:24:53.2696592Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162453.xml
2022-09-27T16:24:53.4890002Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:24:53.5085379Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74039
2022-09-27T16:24:53.5091420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74040
2022-09-27T16:24:54.0167965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:54.0168470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:54.0169460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:54.0169930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:54.0992263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:54.0992792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:54.0997732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:54.0998218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:54.7482721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:54.7483320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:54.7484118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:54.7484825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:54.7491015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:54.7491747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:54.7589235Z STAGE:2022-09-27 16:24:54 73973:73973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:54.7590082Z STAGE:2022-09-27 16:24:54 73974:73974 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:54.7608146Z STAGE:2022-09-27 16:24:54 73973:73973 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:54.7608761Z STAGE:2022-09-27 16:24:54 73974:73974 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:54.7636733Z STAGE:2022-09-27 16:24:54 73974:73974 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:24:54 73973:73973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:54.7637125Z 
2022-09-27T16:24:54.7651269Z STAGE:2022-09-27 16:24:54 73974:73974 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:54.7651866Z STAGE:2022-09-27 16:24:54 73973:73973 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:55.1412126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:55.1412654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:55.1414731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:55.1415244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:55.1951764Z ok (4.410s)
2022-09-27T16:24:55.1952086Z 
2022-09-27T16:24:55.1952651Z ----------------------------------------------------------------------
2022-09-27T16:24:55.1953261Z Ran 1 test in 4.410s
2022-09-27T16:24:55.1953555Z 
2022-09-27T16:24:55.1953714Z OK
2022-09-27T16:24:55.1953961Z 
2022-09-27T16:24:55.1954187Z Generating XML reports...
2022-09-27T16:24:55.1992346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162450.xml
2022-09-27T16:24:55.2032411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:55.2033472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:55.2035862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:55.2036901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:55.2331364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:55.2332265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:55.2333308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:55.2334504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:55.6694835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:24:55.7155591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:24:55.7156415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:55.7201498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:24:55.7209191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:24:55.7310886Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:55.7335378Z 
2022-09-27T16:24:55.7335877Z Running tests...
2022-09-27T16:24:55.7336613Z ----------------------------------------------------------------------
2022-09-27T16:24:55.7357234Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:24:55.7358067Z 
2022-09-27T16:24:55.7358431Z ----------------------------------------------------------------------
2022-09-27T16:24:55.7358760Z Ran 1 test in 0.002s
2022-09-27T16:24:55.7358928Z 
2022-09-27T16:24:55.7359039Z OK (skipped=1)
2022-09-27T16:24:55.7359200Z 
2022-09-27T16:24:55.7359329Z Generating XML reports...
2022-09-27T16:24:55.7393948Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162455.xml
2022-09-27T16:24:55.8170546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:24:57.3599951Z STAGE:2022-09-27 16:24:57 74039:74039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:57.4246760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:57.4247602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:57.4248581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:57.4249326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:57.5110310Z STAGE:2022-09-27 16:24:57 74040:74040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:57.7777784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:57.7778573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:57.7779505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:57.7780278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:24:57.9414827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:24:57.9437957Z 
2022-09-27T16:24:57.9438269Z Running tests...
2022-09-27T16:24:57.9438970Z ----------------------------------------------------------------------
2022-09-27T16:24:57.9458222Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T16:24:57.9458713Z 
2022-09-27T16:24:57.9459154Z ----------------------------------------------------------------------
2022-09-27T16:24:57.9459664Z Ran 1 test in 0.002s
2022-09-27T16:24:57.9459893Z 
2022-09-27T16:24:57.9460056Z OK (skipped=1)
2022-09-27T16:24:57.9460294Z 
2022-09-27T16:24:57.9460483Z Generating XML reports...
2022-09-27T16:24:57.9495785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162457.xml
2022-09-27T16:24:58.2820463Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:24:58.2841425Z 
2022-09-27T16:24:58.2841805Z Running tests...
2022-09-27T16:24:58.2842349Z ----------------------------------------------------------------------
2022-09-27T16:24:58.2867866Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.003s)
2022-09-27T16:24:58.2868188Z 
2022-09-27T16:24:58.2868648Z ----------------------------------------------------------------------
2022-09-27T16:24:58.2869060Z Ran 1 test in 0.003s
2022-09-27T16:24:58.2869230Z 
2022-09-27T16:24:58.2869340Z OK (skipped=1)
2022-09-27T16:24:58.2869497Z 
2022-09-27T16:24:58.2869624Z Generating XML reports...
2022-09-27T16:24:58.2904378Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162458.xml
2022-09-27T16:24:58.8766791Z STAGE:2022-09-27 16:24:58 74039:74039 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:58.8768040Z STAGE:2022-09-27 16:24:58 74040:74040 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:59.0417860Z STAGE:2022-09-27 16:24:59 74040:74040 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:59.0434940Z STAGE:2022-09-27 16:24:59 74040:74040 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:59.0783357Z STAGE:2022-09-27 16:24:59 74039:74039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:24:59.0800697Z STAGE:2022-09-27 16:24:59 74039:74039 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:24:59.9236025Z ok (8.096s)
2022-09-27T16:24:59.9236446Z 
2022-09-27T16:24:59.9237120Z ----------------------------------------------------------------------
2022-09-27T16:24:59.9237771Z Ran 1 test in 8.096s
2022-09-27T16:24:59.9238091Z 
2022-09-27T16:24:59.9238245Z OK
2022-09-27T16:24:59.9238504Z 
2022-09-27T16:24:59.9238749Z Generating XML reports...
2022-09-27T16:24:59.9272527Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162451.xml
2022-09-27T16:24:59.9478676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:24:59.9479650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:24:59.9481646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:24:59.9482572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:00.2898388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:00.2899324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:00.2900527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:00.2901058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:00.4527995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:00.4552083Z 
2022-09-27T16:25:00.4552346Z Running tests...
2022-09-27T16:25:00.4553504Z ----------------------------------------------------------------------
2022-09-27T16:25:00.4573649Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-09-27T16:25:00.4574316Z 
2022-09-27T16:25:00.4574898Z ----------------------------------------------------------------------
2022-09-27T16:25:00.4575531Z Ran 1 test in 0.002s
2022-09-27T16:25:00.4575842Z 
2022-09-27T16:25:00.4576036Z OK (skipped=1)
2022-09-27T16:25:00.4576329Z 
2022-09-27T16:25:00.4576564Z Generating XML reports...
2022-09-27T16:25:00.4613740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162500.xml
2022-09-27T16:25:00.7721997Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:00.7743671Z 
2022-09-27T16:25:00.7743937Z Running tests...
2022-09-27T16:25:00.7744389Z ----------------------------------------------------------------------
2022-09-27T16:25:00.7763506Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:00.7763852Z 
2022-09-27T16:25:00.7764114Z ----------------------------------------------------------------------
2022-09-27T16:25:00.7764455Z Ran 1 test in 0.002s
2022-09-27T16:25:00.7764605Z 
2022-09-27T16:25:00.7764727Z OK (skipped=1)
2022-09-27T16:25:00.7764884Z 
2022-09-27T16:25:00.7765009Z Generating XML reports...
2022-09-27T16:25:00.7798913Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162500.xml
2022-09-27T16:25:02.1467322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:02.1467851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:02.1469263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:02.1470136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:02.5387991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:02.5388507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:02.5389763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:02.5390253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:02.6551732Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:02.6575745Z 
2022-09-27T16:25:02.6575996Z Running tests...
2022-09-27T16:25:02.6576429Z ----------------------------------------------------------------------
2022-09-27T16:25:02.6594474Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:25:02.6594801Z 
2022-09-27T16:25:02.6595065Z ----------------------------------------------------------------------
2022-09-27T16:25:02.6595401Z Ran 1 test in 0.002s
2022-09-27T16:25:02.6595566Z 
2022-09-27T16:25:02.6595678Z OK (skipped=1)
2022-09-27T16:25:02.6595834Z 
2022-09-27T16:25:02.6595962Z Generating XML reports...
2022-09-27T16:25:02.6630837Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162502.xml
2022-09-27T16:25:02.8434809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:02.8435323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:02.8437666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:02.8438156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:03.0289217Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:03.0310888Z 
2022-09-27T16:25:03.0311349Z Running tests...
2022-09-27T16:25:03.0311872Z ----------------------------------------------------------------------
2022-09-27T16:25:03.3351763Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:03.3374436Z 
2022-09-27T16:25:03.3374800Z Running tests...
2022-09-27T16:25:03.3375246Z ----------------------------------------------------------------------
2022-09-27T16:25:03.3394236Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:03.3394636Z 
2022-09-27T16:25:03.3394922Z ----------------------------------------------------------------------
2022-09-27T16:25:03.3395251Z Ran 1 test in 0.002s
2022-09-27T16:25:03.3395417Z 
2022-09-27T16:25:03.3395529Z OK (skipped=1)
2022-09-27T16:25:03.3395687Z 
2022-09-27T16:25:03.3395814Z Generating XML reports...
2022-09-27T16:25:03.3429517Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162503.xml
2022-09-27T16:25:04.6231970Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:04.6426112Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74470
2022-09-27T16:25:04.6432306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74471
2022-09-27T16:25:04.6747500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:04.6748512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:04.6750505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:04.6751849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:05.1797988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:05.1821010Z 
2022-09-27T16:25:05.1821517Z Running tests...
2022-09-27T16:25:05.1822013Z ----------------------------------------------------------------------
2022-09-27T16:25:05.1839067Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:25:05.1839396Z 
2022-09-27T16:25:05.1839693Z ----------------------------------------------------------------------
2022-09-27T16:25:05.1840013Z Ran 1 test in 0.002s
2022-09-27T16:25:05.1840529Z 
2022-09-27T16:25:05.1840658Z OK (skipped=1)
2022-09-27T16:25:05.1840844Z 
2022-09-27T16:25:05.1840971Z Generating XML reports...
2022-09-27T16:25:05.1875980Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162505.xml
2022-09-27T16:25:05.2814085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:05.2815844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:05.2819103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:05.2820290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:05.8112586Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:05.8134736Z 
2022-09-27T16:25:05.8134997Z Running tests...
2022-09-27T16:25:05.8135427Z ----------------------------------------------------------------------
2022-09-27T16:25:05.8153347Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:05.8154063Z 
2022-09-27T16:25:05.8154356Z ----------------------------------------------------------------------
2022-09-27T16:25:05.8154699Z Ran 1 test in 0.002s
2022-09-27T16:25:05.8154848Z 
2022-09-27T16:25:05.8154963Z OK (skipped=1)
2022-09-27T16:25:05.8155120Z 
2022-09-27T16:25:05.8155249Z Generating XML reports...
2022-09-27T16:25:05.8188602Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162505.xml
2022-09-27T16:25:06.3165692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:06.3166511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:06.3168313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:06.3168823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:06.3581449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:06.3582152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:06.3584921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:06.3585423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:06.8839975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:06.8842396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:06.8843602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:06.8844364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:06.8855627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:06.9851325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:06.9948261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:25:06.9959823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:25:06.9961584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:07.0027592Z STAGE:2022-09-27 16:25:07 74470:74470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:07.0052124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:07.0112523Z STAGE:2022-09-27 16:25:07 74471:74471 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:07.0129249Z STAGE:2022-09-27 16:25:07 74471:74471 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:07.0130372Z STAGE:2022-09-27 16:25:07 74470:74470 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:07.0156032Z STAGE:2022-09-27 16:25:07 74471:74471 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:07.0156652Z STAGE:2022-09-27 16:25:07 74470:74470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:07.0171917Z STAGE:2022-09-27 16:25:07 74471:74471 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:07.0173142Z STAGE:2022-09-27 16:25:07 74470:74470 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:07.2554158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:07.2555138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:07.2556305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:07.2557187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:07.4506720Z ok (4.419s)
2022-09-27T16:25:07.4506981Z 
2022-09-27T16:25:07.4507371Z ----------------------------------------------------------------------
2022-09-27T16:25:07.4507710Z Ran 1 test in 4.419s
2022-09-27T16:25:07.4507873Z 
2022-09-27T16:25:07.4507971Z OK
2022-09-27T16:25:07.4508095Z 
2022-09-27T16:25:07.4508236Z Generating XML reports...
2022-09-27T16:25:07.4565002Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162503.xml
2022-09-27T16:25:07.7553310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:07.7576037Z 
2022-09-27T16:25:07.7576629Z Running tests...
2022-09-27T16:25:07.7577228Z ----------------------------------------------------------------------
2022-09-27T16:25:07.8164131Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:07.8164727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:07.8167777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:07.8168287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:08.3454674Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:08.3478089Z 
2022-09-27T16:25:08.3478371Z Running tests...
2022-09-27T16:25:08.3478826Z ----------------------------------------------------------------------
2022-09-27T16:25:08.3498002Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:08.3498666Z 
2022-09-27T16:25:08.3498942Z ----------------------------------------------------------------------
2022-09-27T16:25:08.3499274Z Ran 1 test in 0.002s
2022-09-27T16:25:08.3499441Z 
2022-09-27T16:25:08.3499542Z OK (skipped=1)
2022-09-27T16:25:08.3499683Z 
2022-09-27T16:25:08.3499807Z Generating XML reports...
2022-09-27T16:25:08.3534062Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162508.xml
2022-09-27T16:25:09.4266641Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:09.4432577Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.685s)
2022-09-27T16:25:09.4433928Z 
2022-09-27T16:25:09.4434545Z ----------------------------------------------------------------------
2022-09-27T16:25:09.4435275Z Ran 1 test in 1.685s
2022-09-27T16:25:09.4435628Z 
2022-09-27T16:25:09.4435870Z OK (skipped=1)
2022-09-27T16:25:09.4436234Z 
2022-09-27T16:25:09.4436515Z Generating XML reports...
2022-09-27T16:25:09.4474878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162507.xml
2022-09-27T16:25:09.5207189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:09.5208213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:09.5209975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:09.5210960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:10.0192208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:10.0214738Z 
2022-09-27T16:25:10.0215247Z Running tests...
2022-09-27T16:25:10.0215776Z ----------------------------------------------------------------------
2022-09-27T16:25:10.3506204Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:10.3507310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:10.3508528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:10.3509459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:10.9190901Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:10.9216600Z 
2022-09-27T16:25:10.9216921Z Running tests...
2022-09-27T16:25:10.9217780Z ----------------------------------------------------------------------
2022-09-27T16:25:10.9238256Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:10.9238929Z 
2022-09-27T16:25:10.9239468Z ----------------------------------------------------------------------
2022-09-27T16:25:10.9240116Z Ran 1 test in 0.002s
2022-09-27T16:25:10.9240792Z 
2022-09-27T16:25:10.9240984Z OK (skipped=1)
2022-09-27T16:25:10.9241288Z 
2022-09-27T16:25:10.9241531Z Generating XML reports...
2022-09-27T16:25:10.9277448Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162510.xml
2022-09-27T16:25:11.5110645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:11.5111191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:11.5112063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:11.5112884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:11.6332645Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:11.6533365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74725
2022-09-27T16:25:11.6539220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74726
2022-09-27T16:25:12.0607041Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:12.0628809Z 
2022-09-27T16:25:12.0628967Z Running tests...
2022-09-27T16:25:12.0629795Z ----------------------------------------------------------------------
2022-09-27T16:25:12.9792304Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:12.9793520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:12.9794714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:12.9795682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:13.3788605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:13.3789596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:13.3790777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:13.3791725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:13.4052447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:13.4053473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:13.4055042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:13.4055959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:13.4679186Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:13.4701501Z 
2022-09-27T16:25:13.4701938Z Running tests...
2022-09-27T16:25:13.4702715Z ----------------------------------------------------------------------
2022-09-27T16:25:13.4721672Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:13.4722346Z 
2022-09-27T16:25:13.4722860Z ----------------------------------------------------------------------
2022-09-27T16:25:13.4723486Z Ran 1 test in 0.002s
2022-09-27T16:25:13.4724108Z 
2022-09-27T16:25:13.4724354Z OK (skipped=1)
2022-09-27T16:25:13.4724664Z 
2022-09-27T16:25:13.4724899Z Generating XML reports...
2022-09-27T16:25:13.4757968Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162513.xml
2022-09-27T16:25:13.6667624Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:13.6835919Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2022-09-27T16:25:13.6837094Z 
2022-09-27T16:25:13.6837677Z ----------------------------------------------------------------------
2022-09-27T16:25:13.6838325Z Ran 1 test in 1.620s
2022-09-27T16:25:13.6838656Z 
2022-09-27T16:25:13.6838863Z OK (skipped=1)
2022-09-27T16:25:13.6839156Z 
2022-09-27T16:25:13.6839443Z Generating XML reports...
2022-09-27T16:25:13.6870167Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162512.xml
2022-09-27T16:25:13.9512581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:13.9513640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:13.9514636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:13.9515347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:13.9620828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:14.0526733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:14.3611687Z skip: Skipped due to small world size. (4.339s)
2022-09-27T16:25:14.3612092Z 
2022-09-27T16:25:14.3612599Z ----------------------------------------------------------------------
2022-09-27T16:25:14.3612951Z Ran 1 test in 4.340s
2022-09-27T16:25:14.3613117Z 
2022-09-27T16:25:14.3613225Z OK (skipped=1)
2022-09-27T16:25:14.3613382Z 
2022-09-27T16:25:14.3613508Z Generating XML reports...
2022-09-27T16:25:14.3668226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162510.xml
2022-09-27T16:25:15.5643532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:15.5644053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:15.5647222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:15.5647818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:15.8514896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:15.8515420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:15.8516467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:15.8516941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:16.0671794Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:16.0695261Z 
2022-09-27T16:25:16.0695606Z Running tests...
2022-09-27T16:25:16.0696042Z ----------------------------------------------------------------------
2022-09-27T16:25:16.0714878Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:16.0715255Z 
2022-09-27T16:25:16.0715770Z ----------------------------------------------------------------------
2022-09-27T16:25:16.0716124Z Ran 1 test in 0.002s
2022-09-27T16:25:16.0716289Z 
2022-09-27T16:25:16.0716400Z OK (skipped=1)
2022-09-27T16:25:16.0716567Z 
2022-09-27T16:25:16.0716677Z Generating XML reports...
2022-09-27T16:25:16.0751670Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162516.xml
2022-09-27T16:25:16.3382062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:16.3405101Z 
2022-09-27T16:25:16.3405256Z Running tests...
2022-09-27T16:25:16.3405710Z ----------------------------------------------------------------------
2022-09-27T16:25:16.4506442Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:16.4507042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:16.4508799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:16.4509299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:16.9450700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:16.9472711Z 
2022-09-27T16:25:16.9473025Z Running tests...
2022-09-27T16:25:16.9473477Z ----------------------------------------------------------------------
2022-09-27T16:25:16.9493471Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T16:25:16.9493953Z 
2022-09-27T16:25:16.9494362Z ----------------------------------------------------------------------
2022-09-27T16:25:16.9494705Z Ran 1 test in 0.002s
2022-09-27T16:25:16.9494871Z 
2022-09-27T16:25:16.9494980Z OK (skipped=1)
2022-09-27T16:25:16.9495137Z 
2022-09-27T16:25:16.9495285Z Generating XML reports...
2022-09-27T16:25:16.9529883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162516.xml
2022-09-27T16:25:17.9394520Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:17.9892378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74997
2022-09-27T16:25:17.9984555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74998
2022-09-27T16:25:18.0630330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:18.0630820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:18.0636162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:18.0636871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:18.5739562Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:18.5762828Z 
2022-09-27T16:25:18.5763809Z Running tests...
2022-09-27T16:25:18.5764281Z ----------------------------------------------------------------------
2022-09-27T16:25:18.5781558Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:18.5782144Z 
2022-09-27T16:25:18.5782606Z ----------------------------------------------------------------------
2022-09-27T16:25:18.5782979Z Ran 1 test in 0.002s
2022-09-27T16:25:18.5783156Z 
2022-09-27T16:25:18.5783270Z OK (skipped=1)
2022-09-27T16:25:18.5783412Z 
2022-09-27T16:25:18.5783541Z Generating XML reports...
2022-09-27T16:25:18.5819238Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162518.xml
2022-09-27T16:25:19.0329365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:19.0330461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:19.0331673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:19.0332646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:19.5371219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:19.5395290Z 
2022-09-27T16:25:19.5395600Z Running tests...
2022-09-27T16:25:19.5396293Z ----------------------------------------------------------------------
2022-09-27T16:25:19.5416521Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-09-27T16:25:19.5416871Z 
2022-09-27T16:25:19.5417757Z ----------------------------------------------------------------------
2022-09-27T16:25:19.5418364Z Ran 1 test in 0.002s
2022-09-27T16:25:19.5418525Z 
2022-09-27T16:25:19.5418678Z OK (skipped=1)
2022-09-27T16:25:19.5418833Z 
2022-09-27T16:25:19.5418961Z Generating XML reports...
2022-09-27T16:25:19.5454343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162519.xml
2022-09-27T16:25:19.6474140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:19.6475261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:19.6476623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:19.6477734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:19.7088303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:19.7088783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:19.7089725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:19.7090209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:20.1610404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:20.2184807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:20.2185644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:20.2218703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:20.2226859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:20.3199011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:20.5839594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:20.5841290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:20.5846869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:20.5848047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:21.1104843Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:21.1127179Z 
2022-09-27T16:25:21.1127585Z Running tests...
2022-09-27T16:25:21.1128314Z ----------------------------------------------------------------------
2022-09-27T16:25:21.1146481Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:21.1146816Z 
2022-09-27T16:25:21.1147368Z ----------------------------------------------------------------------
2022-09-27T16:25:21.1147725Z Ran 1 test in 0.002s
2022-09-27T16:25:21.1147900Z 
2022-09-27T16:25:21.1148012Z OK (skipped=1)
2022-09-27T16:25:21.1148187Z 
2022-09-27T16:25:21.1148312Z Generating XML reports...
2022-09-27T16:25:21.1182678Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162521.xml
2022-09-27T16:25:21.5735791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:21.5736302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:21.5737702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:21.5738205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:22.1109943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:22.1135404Z 
2022-09-27T16:25:22.1135867Z Running tests...
2022-09-27T16:25:22.1136943Z ----------------------------------------------------------------------
2022-09-27T16:25:23.0600642Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:23.0601624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:23.0602952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:23.0603442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:23.3106277Z ok (6.970s)
2022-09-27T16:25:23.3106478Z 
2022-09-27T16:25:23.3106859Z ----------------------------------------------------------------------
2022-09-27T16:25:23.3107197Z Ran 1 test in 6.970s
2022-09-27T16:25:23.3107366Z 
2022-09-27T16:25:23.3107484Z OK
2022-09-27T16:25:23.3107622Z 
2022-09-27T16:25:23.3107741Z Generating XML reports...
2022-09-27T16:25:23.3143450Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162516.xml
2022-09-27T16:25:23.5661701Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:23.5685470Z 
2022-09-27T16:25:23.5685769Z Running tests...
2022-09-27T16:25:23.5686212Z ----------------------------------------------------------------------
2022-09-27T16:25:23.8264981Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:23.8471592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75184
2022-09-27T16:25:23.8477580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75185
2022-09-27T16:25:25.2262333Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:25.2461864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75280
2022-09-27T16:25:25.2467213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75281
2022-09-27T16:25:25.4823084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:25.4823584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:25.4826743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:25.4827244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:25.4949557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:25.4950031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:25.4954187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:25.4954702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:25.5765956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:25.5766464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:25.5767781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:25.5768263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:26.0462975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:26.0486671Z 
2022-09-27T16:25:26.0487407Z Running tests...
2022-09-27T16:25:26.0488280Z ----------------------------------------------------------------------
2022-09-27T16:25:26.1000571Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:26.1001285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:26.1002338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:26.1003050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:26.1109809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:26.2015169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:26.6550728Z ok (4.541s)
2022-09-27T16:25:26.6550995Z 
2022-09-27T16:25:26.6551393Z ----------------------------------------------------------------------
2022-09-27T16:25:26.6551734Z Ran 1 test in 4.541s
2022-09-27T16:25:26.6551884Z 
2022-09-27T16:25:26.6551999Z OK
2022-09-27T16:25:26.6552147Z 
2022-09-27T16:25:26.6552284Z Generating XML reports...
2022-09-27T16:25:26.6588170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162522.xml
2022-09-27T16:25:26.8764556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:26.8765082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:26.8767982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:26.8768465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:26.8800000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:26.8800704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:26.8804673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:26.8805148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:27.4003936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:27.4009324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:27.4010142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:27.4105083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:27.4111166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:27.5022897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:27.7064952Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:27.7278522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75362
2022-09-27T16:25:27.7284324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75363
2022-09-27T16:25:27.8536350Z ok (4.284s)
2022-09-27T16:25:27.8536718Z 
2022-09-27T16:25:27.8537375Z ----------------------------------------------------------------------
2022-09-27T16:25:27.8537979Z Ran 1 test in 4.285s
2022-09-27T16:25:27.8538287Z 
2022-09-27T16:25:27.8538462Z OK
2022-09-27T16:25:27.8538716Z 
2022-09-27T16:25:27.8538957Z Generating XML reports...
2022-09-27T16:25:27.8573635Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162523.xml
2022-09-27T16:25:28.7864189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:28.7864917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:28.7866220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:28.7867073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:29.2746820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:29.2768514Z 
2022-09-27T16:25:29.2768944Z Running tests...
2022-09-27T16:25:29.2769779Z ----------------------------------------------------------------------
2022-09-27T16:25:29.3546916Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:29.3548084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:29.3549258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:29.3550323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:29.3726983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:29.3727867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:29.3731370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:29.3732118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:29.8852453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:29.9048908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:29.9049805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:29.9056521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:29.9064765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:29.9133842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:29.9134321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:29.9138052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:29.9138552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:30.0064293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:30.0278639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:25:30.0279706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:25:30.0280930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:30.0281655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:30.4207312Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:30.4231164Z 
2022-09-27T16:25:30.4231329Z Running tests...
2022-09-27T16:25:30.4232599Z ----------------------------------------------------------------------
2022-09-27T16:25:30.4252790Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:30.4253157Z 
2022-09-27T16:25:30.4253449Z ----------------------------------------------------------------------
2022-09-27T16:25:30.4253799Z Ran 1 test in 0.002s
2022-09-27T16:25:30.4253968Z 
2022-09-27T16:25:30.4254080Z OK (skipped=1)
2022-09-27T16:25:30.4254238Z 
2022-09-27T16:25:30.4254366Z Generating XML reports...
2022-09-27T16:25:30.4289170Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162530.xml
2022-09-27T16:25:30.9300103Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:30.9502924Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75514
2022-09-27T16:25:30.9507447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75515
2022-09-27T16:25:32.4763727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:32.4764573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:32.4765572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:32.4766090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:32.6181789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:32.6182327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:32.6184662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:32.6185159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:32.6207809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:32.6208276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:32.6212571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:32.6213075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:32.9873148Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:32.9897037Z 
2022-09-27T16:25:32.9897569Z Running tests...
2022-09-27T16:25:32.9898077Z ----------------------------------------------------------------------
2022-09-27T16:25:32.9916391Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:32.9916751Z 
2022-09-27T16:25:32.9917023Z ----------------------------------------------------------------------
2022-09-27T16:25:32.9917361Z Ran 1 test in 0.002s
2022-09-27T16:25:32.9917525Z 
2022-09-27T16:25:32.9917634Z OK (skipped=1)
2022-09-27T16:25:32.9917791Z 
2022-09-27T16:25:32.9917899Z Generating XML reports...
2022-09-27T16:25:32.9952756Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162532.xml
2022-09-27T16:25:33.1692718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:33.1693334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:33.1694134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:33.1694839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:33.1701265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:33.1701766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:33.2118711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:25:33.2119292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:25:33.2120058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:33.2121072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:33.2268617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:25:33.2269535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:25:33.2270302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:25:33.2370079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:25:33.2495378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-09-27T16:25:33.2496173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-09-27T16:25:33.2497089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-09-27T16:25:33.2598415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-09-27T16:25:33.6694099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:25:33.6694675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:25:33.6695483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:25:33.6696177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:25:33.7454501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-09-27T16:25:33.7455055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-09-27T16:25:33.7455828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-09-27T16:25:33.7456507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-09-27T16:25:33.7574522Z ok (4.480s)
2022-09-27T16:25:33.7574696Z 
2022-09-27T16:25:33.7574995Z ----------------------------------------------------------------------
2022-09-27T16:25:33.7575354Z Ran 1 test in 4.481s
2022-09-27T16:25:33.7575501Z 
2022-09-27T16:25:33.7575601Z OK
2022-09-27T16:25:33.7575735Z 
2022-09-27T16:25:33.7575862Z Generating XML reports...
2022-09-27T16:25:33.7613616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162529.xml
2022-09-27T16:25:34.6436097Z ok (8.595s)
2022-09-27T16:25:34.6436483Z 
2022-09-27T16:25:34.6437166Z ----------------------------------------------------------------------
2022-09-27T16:25:34.6437827Z Ran 1 test in 8.595s
2022-09-27T16:25:34.6438136Z 
2022-09-27T16:25:34.6453339Z OK
2022-09-27T16:25:34.6453676Z 
2022-09-27T16:25:34.6453930Z Generating XML reports...
2022-09-27T16:25:34.6473158Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162526.xml
2022-09-27T16:25:35.0491826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:35.0492793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:35.0494589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:35.5424891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:35.5425576Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:35.5448105Z 
2022-09-27T16:25:35.5448357Z Running tests...
2022-09-27T16:25:35.5449134Z ----------------------------------------------------------------------
2022-09-27T16:25:35.5467930Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:35.5468259Z 
2022-09-27T16:25:35.5468609Z ----------------------------------------------------------------------
2022-09-27T16:25:35.5468931Z Ran 1 test in 0.002s
2022-09-27T16:25:35.5469098Z 
2022-09-27T16:25:35.5469216Z OK (skipped=1)
2022-09-27T16:25:35.5469375Z 
2022-09-27T16:25:35.5469503Z Generating XML reports...
2022-09-27T16:25:35.5503700Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162535.xml
2022-09-27T16:25:35.8855440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:35.8855965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:35.8857470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:35.8857964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:36.3785103Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:36.3809669Z 
2022-09-27T16:25:36.3809963Z Running tests...
2022-09-27T16:25:36.3810411Z ----------------------------------------------------------------------
2022-09-27T16:25:36.3836791Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.003s)
2022-09-27T16:25:36.3837101Z 
2022-09-27T16:25:36.3838159Z ----------------------------------------------------------------------
2022-09-27T16:25:36.3838835Z Ran 1 test in 0.003s
2022-09-27T16:25:36.3839155Z 
2022-09-27T16:25:36.3839357Z OK (skipped=1)
2022-09-27T16:25:36.3839656Z 
2022-09-27T16:25:36.3839901Z Generating XML reports...
2022-09-27T16:25:36.3877238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162536.xml
2022-09-27T16:25:36.7455239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:36.7455753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:36.7457724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:36.7458507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:37.2281097Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:37.2303344Z 
2022-09-27T16:25:37.2303574Z Running tests...
2022-09-27T16:25:37.2304582Z ----------------------------------------------------------------------
2022-09-27T16:25:37.4978212Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:37.4978821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:37.4980158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:37.4980665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:37.9893944Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:37.9916534Z 
2022-09-27T16:25:37.9916896Z Running tests...
2022-09-27T16:25:37.9917484Z ----------------------------------------------------------------------
2022-09-27T16:25:38.3897626Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:38.3898249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:38.3899387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:38.3899868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:38.8465196Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:38.8663548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75850
2022-09-27T16:25:38.8669007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75851
2022-09-27T16:25:38.8984450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:38.9008108Z 
2022-09-27T16:25:38.9008364Z Running tests...
2022-09-27T16:25:38.9008827Z ----------------------------------------------------------------------
2022-09-27T16:25:39.6221565Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:39.6545961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75916
2022-09-27T16:25:39.6693036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75917
2022-09-27T16:25:40.6338952Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:40.6425880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:40.6426792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:40.6428870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:40.6429830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:40.6539404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75982
2022-09-27T16:25:40.6545883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75983
2022-09-27T16:25:40.6962470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:40.6963451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:40.6964588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:40.6965567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:41.1680128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:41.2016532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:41.2017799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:41.2085286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:41.2092915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:41.3048309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:41.4531951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:41.4532474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:41.4533691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:41.4534174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:41.5256433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:41.5256921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:41.5258630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:41.9933459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:41.9933979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:42.0574181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:42.0575075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:42.0645260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:42.0652349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:42.1586229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:42.2918724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:42.2919232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:42.2920848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:42.2921338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:42.3329442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:42.3329949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:42.3331188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:42.3331683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:42.5852966Z ok (4.593s)
2022-09-27T16:25:42.5853331Z 
2022-09-27T16:25:42.5853781Z ----------------------------------------------------------------------
2022-09-27T16:25:42.5854132Z Ran 1 test in 4.593s
2022-09-27T16:25:42.5854283Z 
2022-09-27T16:25:42.5854379Z OK
2022-09-27T16:25:42.5854514Z 
2022-09-27T16:25:42.5854650Z Generating XML reports...
2022-09-27T16:25:42.5912327Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162537.xml
2022-09-27T16:25:42.8861922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:42.8862501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:42.8863519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:42.8864259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:42.8972634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:42.9874984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:42.9986626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:25:43.0010078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:25:43.0010868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:43.0091437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:43.0158443Z STAGE:2022-09-27 16:25:43 75982:75982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:43.0175764Z STAGE:2022-09-27 16:25:43 75983:75983 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:43.0191880Z STAGE:2022-09-27 16:25:43 75982:75982 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:25:43 75983:75983 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:43.0192294Z 
2022-09-27T16:25:43.0218086Z STAGE:2022-09-27 16:25:43 75983:75983 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:25:43 75982:75982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:43.0218845Z 
2022-09-27T16:25:43.0233925Z STAGE:2022-09-27 16:25:43 75982:75982 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:25:43 75983:75983 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:43.0234348Z 
2022-09-27T16:25:43.4614256Z ok (4.560s)
2022-09-27T16:25:43.4614504Z 
2022-09-27T16:25:43.4614901Z ----------------------------------------------------------------------
2022-09-27T16:25:43.4615249Z Ran 1 test in 4.560s
2022-09-27T16:25:43.4615431Z 
2022-09-27T16:25:43.4615527Z OK
2022-09-27T16:25:43.4615650Z 
2022-09-27T16:25:43.4615788Z Generating XML reports...
2022-09-27T16:25:43.4652206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162538.xml
2022-09-27T16:25:44.5812998Z ok (7.350s)
2022-09-27T16:25:44.5813961Z 
2022-09-27T16:25:44.5815074Z ----------------------------------------------------------------------
2022-09-27T16:25:44.5815885Z Ran 1 test in 7.350s
2022-09-27T16:25:44.5820963Z 
2022-09-27T16:25:44.5821575Z OK
2022-09-27T16:25:44.5822177Z 
2022-09-27T16:25:44.5822582Z Generating XML reports...
2022-09-27T16:25:44.5873854Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162537.xml
2022-09-27T16:25:44.8626308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:44.8626837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:44.8629623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:44.8630139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:45.3830662Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:45.3854225Z 
2022-09-27T16:25:45.3854394Z Running tests...
2022-09-27T16:25:45.3855124Z ----------------------------------------------------------------------
2022-09-27T16:25:45.5884617Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:45.5885554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:45.5886877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:45.5887380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:46.1392213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:46.1414708Z 
2022-09-27T16:25:46.1415137Z Running tests...
2022-09-27T16:25:46.1415639Z ----------------------------------------------------------------------
2022-09-27T16:25:46.7178913Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:46.7180030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:46.7181519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:46.7182434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:47.0525398Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:47.0727421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76189
2022-09-27T16:25:47.0733160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76190
2022-09-27T16:25:47.2208562Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:47.2231217Z 
2022-09-27T16:25:47.2231510Z Running tests...
2022-09-27T16:25:47.2231945Z ----------------------------------------------------------------------
2022-09-27T16:25:47.2250403Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:47.2250814Z 
2022-09-27T16:25:47.2251302Z ----------------------------------------------------------------------
2022-09-27T16:25:47.2251648Z Ran 1 test in 0.002s
2022-09-27T16:25:47.2251798Z 
2022-09-27T16:25:47.2251912Z OK (skipped=1)
2022-09-27T16:25:47.2252071Z 
2022-09-27T16:25:47.2252209Z Generating XML reports...
2022-09-27T16:25:47.2286086Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162547.xml
2022-09-27T16:25:47.7187521Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:47.7796444Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76255
2022-09-27T16:25:47.7968684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76256
2022-09-27T16:25:48.8778092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:48.8778614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:48.8780477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:48.8780979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:48.9042137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:48.9042619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:48.9045518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:48.9046012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:49.3002142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:49.3002639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:49.3004647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:49.3005436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:49.4002555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:49.4088350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:49.4089162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:49.4103522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:49.4110314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:49.4113647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:25:49.5101652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:49.5104294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:25:49.5105070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:49.5127224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:49.5169832Z STAGE:2022-09-27 16:25:49 76189:76189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:49.5186195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:49.5186682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:49.5187240Z STAGE:2022-09-27 16:25:49 76190:76190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:49.5190504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:49.5191006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:49.5796847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:49.5797367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:49.5798911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:49.5799401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:49.7917676Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:49.7940164Z 
2022-09-27T16:25:49.7940490Z Running tests...
2022-09-27T16:25:49.7940961Z ----------------------------------------------------------------------
2022-09-27T16:25:49.7959616Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:49.7959972Z 
2022-09-27T16:25:49.7960549Z ----------------------------------------------------------------------
2022-09-27T16:25:49.7960887Z Ran 1 test in 0.002s
2022-09-27T16:25:49.7961056Z 
2022-09-27T16:25:49.7961166Z OK (skipped=1)
2022-09-27T16:25:49.7961325Z 
2022-09-27T16:25:49.7961453Z Generating XML reports...
2022-09-27T16:25:49.7995375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162549.xml
2022-09-27T16:25:50.1498854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:50.1499425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:50.1500217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:50.1501129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:50.1609064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:50.2513502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:50.2624686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:25:50.2625220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:25:50.2626005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:50.2626853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:50.2728703Z STAGE:2022-09-27 16:25:50 76256:76256 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:25:50 76255:76255 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:50.2729436Z 
2022-09-27T16:25:50.2743356Z STAGE:2022-09-27 16:25:50 76255:76255 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:50.2744265Z STAGE:2022-09-27 16:25:50 76256:76256 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:50.2769733Z STAGE:2022-09-27 16:25:50 76255:76255 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:25:50 76256:76256 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:50.2770123Z 
2022-09-27T16:25:50.2784125Z STAGE:2022-09-27 16:25:50 76255:76255 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:50.2784732Z STAGE:2022-09-27 16:25:50 76256:76256 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:50.7039084Z ok (4.562s)
2022-09-27T16:25:50.7039309Z 
2022-09-27T16:25:50.7039728Z ----------------------------------------------------------------------
2022-09-27T16:25:50.7040076Z Ran 1 test in 4.562s
2022-09-27T16:25:50.7040608Z 
2022-09-27T16:25:50.7040715Z OK
2022-09-27T16:25:50.7040868Z 
2022-09-27T16:25:50.7042245Z Generating XML reports...
2022-09-27T16:25:50.7076038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162546.xml
2022-09-27T16:25:51.1520114Z [1664295951.151508] [28fd6574525b:76189:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:25:51.1521245Z [1664295951.151508] [28fd6574525b:76189:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:25:51.1894152Z [1664295951.188948] [28fd6574525b:76190:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:25:51.1894693Z [1664295951.188948] [28fd6574525b:76190:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:25:51.5974008Z STAGE:2022-09-27 16:25:51 76190:76190 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:25:51 76189:76189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:51.5974466Z 
2022-09-27T16:25:51.7209214Z STAGE:2022-09-27 16:25:51 76190:76190 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:51.7225311Z STAGE:2022-09-27 16:25:51 76189:76189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:51.7235342Z STAGE:2022-09-27 16:25:51 76190:76190 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:51.7235932Z STAGE:2022-09-27 16:25:51 76189:76189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:51.7969856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:51.7970669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:51.7972834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:51.7973343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:52.3061692Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:52.3086453Z 
2022-09-27T16:25:52.3086967Z Running tests...
2022-09-27T16:25:52.3087485Z ----------------------------------------------------------------------
2022-09-27T16:25:52.3106746Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:52.3107386Z 
2022-09-27T16:25:52.3107984Z ----------------------------------------------------------------------
2022-09-27T16:25:52.3108693Z Ran 1 test in 0.002s
2022-09-27T16:25:52.3109022Z 
2022-09-27T16:25:52.3109233Z OK (skipped=1)
2022-09-27T16:25:52.3109534Z 
2022-09-27T16:25:52.3109767Z Generating XML reports...
2022-09-27T16:25:52.3147336Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162552.xml
2022-09-27T16:25:52.8388752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:52.8389316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:52.8391690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:52.8392188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:52.9857962Z ok (7.600s)
2022-09-27T16:25:52.9858337Z 
2022-09-27T16:25:52.9859090Z ----------------------------------------------------------------------
2022-09-27T16:25:52.9859473Z Ran 1 test in 7.600s
2022-09-27T16:25:52.9859641Z 
2022-09-27T16:25:52.9859744Z OK
2022-09-27T16:25:52.9859873Z 
2022-09-27T16:25:52.9860006Z Generating XML reports...
2022-09-27T16:25:52.9894858Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162545.xml
2022-09-27T16:25:53.4089117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:25:53.4121230Z 
2022-09-27T16:25:53.4121453Z Running tests...
2022-09-27T16:25:53.4121887Z ----------------------------------------------------------------------
2022-09-27T16:25:54.3204432Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:54.3205071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:54.3206742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:54.3207293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:54.8279701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:54.8302828Z 
2022-09-27T16:25:54.8303115Z Running tests...
2022-09-27T16:25:54.8303815Z ----------------------------------------------------------------------
2022-09-27T16:25:54.8322664Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:54.8323029Z 
2022-09-27T16:25:54.8323440Z ----------------------------------------------------------------------
2022-09-27T16:25:54.8323911Z Ran 1 test in 0.002s
2022-09-27T16:25:54.8324082Z 
2022-09-27T16:25:54.8324193Z OK (skipped=1)
2022-09-27T16:25:54.8324352Z 
2022-09-27T16:25:54.8324461Z Generating XML reports...
2022-09-27T16:25:54.8358683Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162554.xml
2022-09-27T16:25:55.0706640Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:55.0900526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76521
2022-09-27T16:25:55.0906281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76522
2022-09-27T16:25:55.1408181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:55.1408692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:55.1411268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:55.1412018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:55.6284022Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:25:55.6306422Z 
2022-09-27T16:25:55.6306570Z Running tests...
2022-09-27T16:25:55.6307346Z ----------------------------------------------------------------------
2022-09-27T16:25:56.7481009Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:56.7481917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:56.7482547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:56.7483013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:56.7483600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:56.7484069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:56.7484666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:56.7485156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:56.7517023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:56.7517484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:56.7521586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:56.7522076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:57.2257488Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:25:57.2450018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76623
2022-09-27T16:25:57.2456098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76624
2022-09-27T16:25:57.2622158Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:57.2646284Z 
2022-09-27T16:25:57.2646448Z Running tests...
2022-09-27T16:25:57.2647371Z ----------------------------------------------------------------------
2022-09-27T16:25:57.2666081Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:57.2666463Z 
2022-09-27T16:25:57.2666749Z ----------------------------------------------------------------------
2022-09-27T16:25:57.2667250Z Ran 1 test in 0.002s
2022-09-27T16:25:57.2667529Z 
2022-09-27T16:25:57.2667642Z OK (skipped=1)
2022-09-27T16:25:57.2667789Z 
2022-09-27T16:25:57.2667919Z Generating XML reports...
2022-09-27T16:25:57.2701461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162557.xml
2022-09-27T16:25:57.3130187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:57.3130978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:57.3131748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:57.3132463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:57.3138541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:57.3139533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:57.3248551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:25:57.3249103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:25:57.3249903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:57.3250786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:57.3346505Z STAGE:2022-09-27 16:25:57 76522:76522 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:25:57 76521:76521 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:57.3347206Z 
2022-09-27T16:25:57.3363330Z STAGE:2022-09-27 16:25:57 76521:76521 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:25:57 76522:76522 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:57.3363807Z 
2022-09-27T16:25:57.3390344Z STAGE:2022-09-27 16:25:57 76522:76522 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:25:57 76521:76521 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:57.3390985Z 
2022-09-27T16:25:57.3406201Z STAGE:2022-09-27 16:25:57 76522:76522 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:57.3407125Z STAGE:2022-09-27 16:25:57 76521:76521 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:25:57.7972546Z ok (4.385s)
2022-09-27T16:25:57.7972751Z 
2022-09-27T16:25:57.7973176Z ----------------------------------------------------------------------
2022-09-27T16:25:57.7973524Z Ran 1 test in 4.385s
2022-09-27T16:25:57.7973692Z 
2022-09-27T16:25:57.7973796Z OK
2022-09-27T16:25:57.7973931Z 
2022-09-27T16:25:57.7974047Z Generating XML reports...
2022-09-27T16:25:57.8011880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162553.xml
2022-09-27T16:25:58.9316994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:58.9317986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:58.9319195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:58.9320131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:58.9383678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:58.9384587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:58.9387488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:58.9388362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:59.3101546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:59.3102526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:59.3103694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:59.3105069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:25:59.5022137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:25:59.5071085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:25:59.5072603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:59.5123881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:25:59.5130490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:25:59.5133684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:25:59.6084361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:25:59.6088134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:25:59.6089529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:59.6149294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:25:59.6156356Z STAGE:2022-09-27 16:25:59 76623:76623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:59.6207894Z STAGE:2022-09-27 16:25:59 76624:76624 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:25:59.7974921Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:25:59.7997332Z 
2022-09-27T16:25:59.7997772Z Running tests...
2022-09-27T16:25:59.7998628Z ----------------------------------------------------------------------
2022-09-27T16:25:59.8018854Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:25:59.8019551Z 
2022-09-27T16:25:59.8020081Z ----------------------------------------------------------------------
2022-09-27T16:25:59.8020765Z Ran 1 test in 0.002s
2022-09-27T16:25:59.8021112Z 
2022-09-27T16:25:59.8021292Z OK (skipped=1)
2022-09-27T16:25:59.8021614Z 
2022-09-27T16:25:59.8021886Z Generating XML reports...
2022-09-27T16:25:59.8056126Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162559.xml
2022-09-27T16:25:59.9956389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:25:59.9957396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:25:59.9958628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:25:59.9959608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:00.5172585Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:00.5195393Z 
2022-09-27T16:26:00.5195830Z Running tests...
2022-09-27T16:26:00.5196665Z ----------------------------------------------------------------------
2022-09-27T16:26:01.2945460Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... [1664295961.293888] [28fd6574525b:76623:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:26:01.2946898Z [1664295961.293888] [28fd6574525b:76623:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:26:01.3213769Z [1664295961.320913] [28fd6574525b:76624:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:26:01.3214538Z [1664295961.320913] [28fd6574525b:76624:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:26:01.7505842Z STAGE:2022-09-27 16:26:01 76624:76624 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:26:01 76623:76623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:01.7506740Z 
2022-09-27T16:26:01.7550754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:01.7551687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:01.7554197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:01.7555188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:01.8789564Z STAGE:2022-09-27 16:26:01 76624:76624 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:01.8822509Z STAGE:2022-09-27 16:26:01 76623:76623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:01.8832626Z STAGE:2022-09-27 16:26:01 76624:76624 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:01.8834250Z STAGE:2022-09-27 16:26:01 76623:76623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:02.1909848Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:02.2120090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76823
2022-09-27T16:26:02.2125499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76824
2022-09-27T16:26:02.2831689Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:02.2854712Z 
2022-09-27T16:26:02.2854858Z Running tests...
2022-09-27T16:26:02.2855960Z ----------------------------------------------------------------------
2022-09-27T16:26:02.2875057Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:26:02.2875433Z 
2022-09-27T16:26:02.2875710Z ----------------------------------------------------------------------
2022-09-27T16:26:02.2876050Z Ran 1 test in 0.002s
2022-09-27T16:26:02.2876229Z 
2022-09-27T16:26:02.2876338Z OK (skipped=1)
2022-09-27T16:26:02.2876497Z 
2022-09-27T16:26:02.2876606Z Generating XML reports...
2022-09-27T16:26:02.2911397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162602.xml
2022-09-27T16:26:03.1593031Z ok (7.528s)
2022-09-27T16:26:03.1593452Z 
2022-09-27T16:26:03.1594109Z ----------------------------------------------------------------------
2022-09-27T16:26:03.1594754Z Ran 1 test in 7.528s
2022-09-27T16:26:03.1595071Z 
2022-09-27T16:26:03.1595235Z OK
2022-09-27T16:26:03.1595517Z 
2022-09-27T16:26:03.1595739Z Generating XML reports...
2022-09-27T16:26:03.1630887Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162555.xml
2022-09-27T16:26:03.9262846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:03.9263850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:03.9265053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:03.9266006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:03.9288858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:03.9289805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:03.9294132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:03.9295104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:04.2948196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:04.2949194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:04.2950444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:04.2951400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:04.5180498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:04.5181372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:04.5182794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:04.5184171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:04.5188245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:04.5189978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:04.5297950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:26:04.5298499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:26:04.5299219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:26:04.5300076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:26:04.5365452Z STAGE:2022-09-27 16:26:04 76823:76823 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:04.5460701Z STAGE:2022-09-27 16:26:04 76824:76824 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:04.5479768Z STAGE:2022-09-27 16:26:04 76823:76823 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:26:04 76824:76824 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:04.5480878Z 
2022-09-27T16:26:04.5508822Z STAGE:2022-09-27 16:26:04 76823:76823 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:04.5509454Z STAGE:2022-09-27 16:26:04 76824:76824 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:04.5524269Z STAGE:2022-09-27 16:26:04 76824:76824 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:04.5524879Z STAGE:2022-09-27 16:26:04 76823:76823 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:04.8028859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:04.8053052Z 
2022-09-27T16:26:04.8053503Z Running tests...
2022-09-27T16:26:04.8053972Z ----------------------------------------------------------------------
2022-09-27T16:26:04.8073404Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:26:04.8074181Z 
2022-09-27T16:26:04.8074474Z ----------------------------------------------------------------------
2022-09-27T16:26:04.8074826Z Ran 1 test in 0.002s
2022-09-27T16:26:04.8074994Z 
2022-09-27T16:26:04.8075087Z OK (skipped=1)
2022-09-27T16:26:04.8075246Z 
2022-09-27T16:26:04.8075379Z Generating XML reports...
2022-09-27T16:26:04.8111733Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162604.xml
2022-09-27T16:26:05.0203389Z ok (4.500s)
2022-09-27T16:26:05.0203608Z 
2022-09-27T16:26:05.0203992Z ----------------------------------------------------------------------
2022-09-27T16:26:05.0204336Z Ran 1 test in 4.501s
2022-09-27T16:26:05.0204510Z 
2022-09-27T16:26:05.0204838Z OK
2022-09-27T16:26:05.0204996Z 
2022-09-27T16:26:05.0205134Z Generating XML reports...
2022-09-27T16:26:05.0256690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162600.xml
2022-09-27T16:26:05.2327472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:05.2327962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:05.2329547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:05.2330038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:05.7233294Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:26:05.7255886Z 
2022-09-27T16:26:05.7256116Z Running tests...
2022-09-27T16:26:05.7256599Z ----------------------------------------------------------------------
2022-09-27T16:26:06.8326599Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:06.8327850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:06.8329068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:06.8329604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:07.1266596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:07.1267073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:07.1269085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:07.1269602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:07.3299697Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:07.3359779Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:07.3384840Z 
2022-09-27T16:26:07.3385147Z Running tests...
2022-09-27T16:26:07.3385590Z ----------------------------------------------------------------------
2022-09-27T16:26:07.3404865Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:26:07.3405194Z 
2022-09-27T16:26:07.3405445Z ----------------------------------------------------------------------
2022-09-27T16:26:07.3405783Z Ran 1 test in 0.002s
2022-09-27T16:26:07.3405955Z 
2022-09-27T16:26:07.3406070Z OK (skipped=1)
2022-09-27T16:26:07.3406228Z 
2022-09-27T16:26:07.3406355Z Generating XML reports...
2022-09-27T16:26:07.3440938Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162607.xml
2022-09-27T16:26:07.3494582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77041
2022-09-27T16:26:07.3500830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77042
2022-09-27T16:26:07.6179752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:07.6202075Z 
2022-09-27T16:26:07.6202306Z Running tests...
2022-09-27T16:26:07.6202743Z ----------------------------------------------------------------------
2022-09-27T16:26:08.9957258Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:08.9957863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:08.9958698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:08.9959512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:09.0523796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:09.0524293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:09.0526832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:09.0527331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:09.2438881Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:09.2633263Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77141
2022-09-27T16:26:09.2639879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77142
2022-09-27T16:26:09.3474596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:09.3475143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:09.3476238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:09.3477150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:09.4990605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:09.5748422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:09.5749234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:09.5800934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:09.5808322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:09.5811654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:26:09.6761740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:09.6764402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:26:09.6765219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:26:09.6826586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:26:09.6831001Z STAGE:2022-09-27 16:26:09 77041:77041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:09.6886408Z STAGE:2022-09-27 16:26:09 77042:77042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:09.8744165Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:09.8783888Z 
2022-09-27T16:26:09.8785951Z Running tests...
2022-09-27T16:26:09.8787724Z ----------------------------------------------------------------------
2022-09-27T16:26:09.8818121Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-09-27T16:26:09.8820087Z 
2022-09-27T16:26:09.8821353Z ----------------------------------------------------------------------
2022-09-27T16:26:09.8822068Z Ran 1 test in 0.003s
2022-09-27T16:26:09.8822423Z 
2022-09-27T16:26:09.8822666Z OK (skipped=1)
2022-09-27T16:26:09.8823005Z 
2022-09-27T16:26:09.8823269Z Generating XML reports...
2022-09-27T16:26:09.8875528Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162609.xml
2022-09-27T16:26:10.9033357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:10.9034191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:10.9035209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:10.9035720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:10.9860195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:10.9861029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:10.9866299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:10.9867223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:11.3747509Z [1664295971.374213] [28fd6574525b:77042:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:26:11.3748069Z [1664295971.374213] [28fd6574525b:77042:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:26:11.4012353Z [1664295971.400763] [28fd6574525b:77041:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:26:11.4013135Z [1664295971.400763] [28fd6574525b:77041:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:26:11.5530690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:11.5531274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:11.5532090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:11.5532811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:11.5539305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:11.5539899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:11.7782704Z STAGE:2022-09-27 16:26:11 77042:77042 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:26:11 77041:77041 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:11.7783157Z 
2022-09-27T16:26:11.9076116Z STAGE:2022-09-27 16:26:11 77041:77041 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:11.9081607Z STAGE:2022-09-27 16:26:11 77042:77042 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:11.9091882Z STAGE:2022-09-27 16:26:11 77041:77041 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:11.9092499Z STAGE:2022-09-27 16:26:11 77042:77042 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:11.9656590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:11.9657099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:11.9658412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:11.9658888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:11.9711253Z skip: Skipped due to small world size. (4.351s)
2022-09-27T16:26:11.9711986Z 
2022-09-27T16:26:11.9712293Z ----------------------------------------------------------------------
2022-09-27T16:26:11.9712647Z Ran 1 test in 4.351s
2022-09-27T16:26:11.9712797Z 
2022-09-27T16:26:11.9713289Z OK (skipped=1)
2022-09-27T16:26:11.9713497Z 
2022-09-27T16:26:11.9713632Z Generating XML reports...
2022-09-27T16:26:11.9772647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162607.xml
2022-09-27T16:26:12.4752558Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:12.4775397Z 
2022-09-27T16:26:12.4775862Z Running tests...
2022-09-27T16:26:12.4776350Z ----------------------------------------------------------------------
2022-09-27T16:26:12.4794204Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:26:12.4794547Z 
2022-09-27T16:26:12.4794809Z ----------------------------------------------------------------------
2022-09-27T16:26:12.4795127Z Ran 1 test in 0.002s
2022-09-27T16:26:12.4795294Z 
2022-09-27T16:26:12.4795407Z OK (skipped=1)
2022-09-27T16:26:12.4795565Z 
2022-09-27T16:26:12.4795692Z Generating XML reports...
2022-09-27T16:26:12.4829600Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162612.xml
2022-09-27T16:26:13.1644887Z ok (7.438s)
2022-09-27T16:26:13.1648093Z 
2022-09-27T16:26:13.1650800Z ----------------------------------------------------------------------
2022-09-27T16:26:13.1654256Z Ran 1 test in 7.438s
2022-09-27T16:26:13.1656911Z 
2022-09-27T16:26:13.1659911Z OK
2022-09-27T16:26:13.1662259Z 
2022-09-27T16:26:13.1664444Z Generating XML reports...
2022-09-27T16:26:13.1724270Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162605.xml
2022-09-27T16:26:14.1766002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:14.1766516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:14.1767933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:14.1768404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:14.4808455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:14.4809061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:14.4811937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:14.4812437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:14.7374454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:14.7398008Z 
2022-09-27T16:26:14.7398259Z Running tests...
2022-09-27T16:26:14.7398678Z ----------------------------------------------------------------------
2022-09-27T16:26:14.9643769Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:14.9666007Z 
2022-09-27T16:26:14.9666444Z Running tests...
2022-09-27T16:26:14.9666977Z ----------------------------------------------------------------------
2022-09-27T16:26:14.9686474Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:26:14.9686831Z 
2022-09-27T16:26:14.9687086Z ----------------------------------------------------------------------
2022-09-27T16:26:14.9687420Z Ran 1 test in 0.002s
2022-09-27T16:26:14.9687590Z 
2022-09-27T16:26:14.9687704Z OK (skipped=1)
2022-09-27T16:26:14.9687865Z 
2022-09-27T16:26:14.9687998Z Generating XML reports...
2022-09-27T16:26:14.9722128Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162614.xml
2022-09-27T16:26:15.3032487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:15.3033013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:15.3035112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:15.3035633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:15.8012838Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:26:15.8035384Z 
2022-09-27T16:26:15.8035656Z Running tests...
2022-09-27T16:26:15.8036077Z ----------------------------------------------------------------------
2022-09-27T16:26:16.3590963Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:16.3786827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77398
2022-09-27T16:26:16.3793723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77399
2022-09-27T16:26:16.8439734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:16.8440741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:16.8441859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:16.8442664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:17.3947509Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:17.3969830Z 
2022-09-27T16:26:17.3970264Z Running tests...
2022-09-27T16:26:17.3970780Z ----------------------------------------------------------------------
2022-09-27T16:26:17.3988371Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:26:17.3988716Z 
2022-09-27T16:26:17.3988990Z ----------------------------------------------------------------------
2022-09-27T16:26:17.3989327Z Ran 1 test in 0.002s
2022-09-27T16:26:17.3989496Z 
2022-09-27T16:26:17.3989617Z OK (skipped=1)
2022-09-27T16:26:17.3989796Z 
2022-09-27T16:26:17.3989908Z Generating XML reports...
2022-09-27T16:26:17.4024156Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162617.xml
2022-09-27T16:26:17.4270309Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:17.4474642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77464
2022-09-27T16:26:17.4480287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77465
2022-09-27T16:26:18.0044069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:18.0045053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:18.0046717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:18.0047663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:18.0326883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:18.0327837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:18.0329827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:18.0330792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:18.6735631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:18.6736202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:18.6737012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:18.6737926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:18.6744316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:18.6744818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:19.0859172Z skip: Skipped due to small world size. (4.346s)
2022-09-27T16:26:19.0859409Z 
2022-09-27T16:26:19.0859821Z ----------------------------------------------------------------------
2022-09-27T16:26:19.0860168Z Ran 1 test in 4.346s
2022-09-27T16:26:19.0860335Z 
2022-09-27T16:26:19.0860449Z OK (skipped=1)
2022-09-27T16:26:19.0860607Z 
2022-09-27T16:26:19.0860717Z Generating XML reports...
2022-09-27T16:26:19.0894849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162614.xml
2022-09-27T16:26:19.1523453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:19.1524315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:19.1525830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:19.1526640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:19.1539967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:19.1540444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:19.1544848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:19.1545347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:19.3260736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:19.3261346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:19.3264182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:19.3264681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:19.6689741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:19.6819226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:19.6820288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:19.6891814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:19.6898335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:19.6901512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:26:19.7831039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:19.7834286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:26:19.7835036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:26:19.7899841Z STAGE:2022-09-27 16:26:19 77465:77465 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:19.7914983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:26:19.7973177Z STAGE:2022-09-27 16:26:19 77464:77464 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:19.8143613Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:19.8167596Z 
2022-09-27T16:26:19.8168028Z Running tests...
2022-09-27T16:26:19.8168504Z ----------------------------------------------------------------------
2022-09-27T16:26:19.8189888Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:19.8190490Z 
2022-09-27T16:26:19.8190916Z ----------------------------------------------------------------------
2022-09-27T16:26:19.8191288Z Ran 1 test in 0.002s
2022-09-27T16:26:19.8191439Z 
2022-09-27T16:26:19.8191553Z OK (skipped=1)
2022-09-27T16:26:19.8191714Z 
2022-09-27T16:26:19.8191843Z Generating XML reports...
2022-09-27T16:26:19.8227640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162619.xml
2022-09-27T16:26:21.2111348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:21.2111868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:21.2114302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:21.2114809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:21.4989042Z [1664295981.498363] [28fd6574525b:77464:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:26:21.4989568Z [1664295981.498363] [28fd6574525b:77464:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:26:21.5397402Z [1664295981.539192] [28fd6574525b:77465:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:26:21.5397962Z [1664295981.539192] [28fd6574525b:77465:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:26:21.7310069Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:21.7334598Z 
2022-09-27T16:26:21.7334916Z Running tests...
2022-09-27T16:26:21.7335376Z ----------------------------------------------------------------------
2022-09-27T16:26:21.8033017Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:21.8033642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:21.8034547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:21.8035181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:21.9321660Z STAGE:2022-09-27 16:26:21 77464:77464 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:26:21 77465:77465 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:21.9322110Z 
2022-09-27T16:26:22.0569302Z STAGE:2022-09-27 16:26:22 77465:77465 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:22.0575609Z STAGE:2022-09-27 16:26:22 77464:77464 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:22.0586302Z STAGE:2022-09-27 16:26:22 77465:77465 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:26:22 77464:77464 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:22.0586720Z 
2022-09-27T16:26:22.3293584Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:22.3317224Z 
2022-09-27T16:26:22.3317605Z Running tests...
2022-09-27T16:26:22.3318237Z ----------------------------------------------------------------------
2022-09-27T16:26:22.3336473Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:22.3336791Z 
2022-09-27T16:26:22.3337719Z ----------------------------------------------------------------------
2022-09-27T16:26:22.3338083Z Ran 1 test in 0.002s
2022-09-27T16:26:22.3338256Z 
2022-09-27T16:26:22.3338368Z OK (skipped=1)
2022-09-27T16:26:22.3338524Z 
2022-09-27T16:26:22.3338661Z Generating XML reports...
2022-09-27T16:26:22.3371618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162622.xml
2022-09-27T16:26:23.2638992Z ok (7.459s)
2022-09-27T16:26:23.2643506Z 
2022-09-27T16:26:23.2647427Z ----------------------------------------------------------------------
2022-09-27T16:26:23.2649198Z Ran 1 test in 7.459s
2022-09-27T16:26:23.2649988Z 
2022-09-27T16:26:23.2650223Z OK
2022-09-27T16:26:23.2650538Z 
2022-09-27T16:26:23.2651206Z Generating XML reports...
2022-09-27T16:26:23.2704462Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162615.xml
2022-09-27T16:26:23.3937555Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:23.4145984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77688
2022-09-27T16:26:23.4152719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77689
2022-09-27T16:26:24.3703646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:24.3704682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:24.3706286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:24.3707214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:24.9268878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:24.9291937Z 
2022-09-27T16:26:24.9292252Z Running tests...
2022-09-27T16:26:24.9293129Z ----------------------------------------------------------------------
2022-09-27T16:26:24.9311962Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:24.9312609Z 
2022-09-27T16:26:24.9313159Z ----------------------------------------------------------------------
2022-09-27T16:26:24.9313794Z Ran 1 test in 0.002s
2022-09-27T16:26:24.9314109Z 
2022-09-27T16:26:24.9314311Z OK (skipped=1)
2022-09-27T16:26:24.9314611Z 
2022-09-27T16:26:24.9314849Z Generating XML reports...
2022-09-27T16:26:24.9348815Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162624.xml
2022-09-27T16:26:25.0106573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:25.0107566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:25.0108735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:25.0109725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:25.0813691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:25.0814694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:25.0815840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:25.0816752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:25.4397077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:25.4398077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:25.4399910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:25.4401232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:25.5972359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:25.5973472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:25.5974945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:25.5976376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:25.6079612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:25.6986339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:25.9328653Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:26:25.9352228Z 
2022-09-27T16:26:25.9353099Z Running tests...
2022-09-27T16:26:25.9354216Z ----------------------------------------------------------------------
2022-09-27T16:26:26.0222536Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Skipped due to small world size. (4.288s)
2022-09-27T16:26:26.0223221Z 
2022-09-27T16:26:26.0223603Z ----------------------------------------------------------------------
2022-09-27T16:26:26.0223941Z Ran 1 test in 4.289s
2022-09-27T16:26:26.0224107Z 
2022-09-27T16:26:26.0224200Z OK (skipped=1)
2022-09-27T16:26:26.0224358Z 
2022-09-27T16:26:26.0224482Z Generating XML reports...
2022-09-27T16:26:26.0259654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162621.xml
2022-09-27T16:26:26.8859022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:26.8859532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:26.8860340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:26.8860818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:27.4525719Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:27.4548552Z 
2022-09-27T16:26:27.4549027Z Running tests...
2022-09-27T16:26:27.4549526Z ----------------------------------------------------------------------
2022-09-27T16:26:27.4569883Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:27.4570467Z 
2022-09-27T16:26:27.4571082Z ----------------------------------------------------------------------
2022-09-27T16:26:27.4571637Z Ran 1 test in 0.002s
2022-09-27T16:26:27.4571804Z 
2022-09-27T16:26:27.4571913Z OK (skipped=1)
2022-09-27T16:26:27.4572072Z 
2022-09-27T16:26:27.4572199Z Generating XML reports...
2022-09-27T16:26:27.4606305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162627.xml
2022-09-27T16:26:27.6324966Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:27.6522071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77863
2022-09-27T16:26:27.6528395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77864
2022-09-27T16:26:28.1334122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:28.1334633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:28.1335810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:28.1336288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:28.7512946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:28.7534642Z 
2022-09-27T16:26:28.7535593Z Running tests...
2022-09-27T16:26:28.7536507Z ----------------------------------------------------------------------
2022-09-27T16:26:29.3116867Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:29.3118050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:29.3119184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:29.3120060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:29.3310470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:29.3311419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:29.3313840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:29.3314772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:29.4079027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:29.4079894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:29.4081263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:29.4082102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:29.8244173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:29.8401155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:29.8402659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:29.8446642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:29.8453705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:29.9087974Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:29.9110869Z 
2022-09-27T16:26:29.9111314Z Running tests...
2022-09-27T16:26:29.9111820Z ----------------------------------------------------------------------
2022-09-27T16:26:29.9132333Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:29.9132664Z 
2022-09-27T16:26:29.9132946Z ----------------------------------------------------------------------
2022-09-27T16:26:29.9133281Z Ran 1 test in 0.002s
2022-09-27T16:26:29.9133450Z 
2022-09-27T16:26:29.9133564Z OK (skipped=1)
2022-09-27T16:26:29.9133721Z 
2022-09-27T16:26:29.9133850Z Generating XML reports...
2022-09-27T16:26:29.9169608Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162629.xml
2022-09-27T16:26:29.9416045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:30.3029399Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:30.3223716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77968
2022-09-27T16:26:30.3229399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77969
2022-09-27T16:26:30.3591234Z skip: Skipped due to small world size. (4.424s)
2022-09-27T16:26:30.3591465Z 
2022-09-27T16:26:30.3592366Z ----------------------------------------------------------------------
2022-09-27T16:26:30.3592740Z Ran 1 test in 4.424s
2022-09-27T16:26:30.3592913Z 
2022-09-27T16:26:30.3593029Z OK (skipped=1)
2022-09-27T16:26:30.3593196Z 
2022-09-27T16:26:30.3593590Z Generating XML reports...
2022-09-27T16:26:30.3629440Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162625.xml
2022-09-27T16:26:31.9825112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:31.9825631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:31.9826910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:31.9827401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:32.0464705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:32.0465204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:32.0467580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:32.0468072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:32.0647401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:32.0648246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:32.0650673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:32.0651162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:32.5622179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:32.5622693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:32.5625530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:32.5626025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:32.5634554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:32.5660364Z 
2022-09-27T16:26:32.5660777Z Running tests...
2022-09-27T16:26:32.5661289Z ----------------------------------------------------------------------
2022-09-27T16:26:32.5679884Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:32.5680499Z 
2022-09-27T16:26:32.5680917Z ----------------------------------------------------------------------
2022-09-27T16:26:32.5681260Z Ran 1 test in 0.002s
2022-09-27T16:26:32.5681426Z 
2022-09-27T16:26:32.5681534Z OK (skipped=1)
2022-09-27T16:26:32.5681698Z 
2022-09-27T16:26:32.5681828Z Generating XML reports...
2022-09-27T16:26:32.5710816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:32.5711831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:32.5713113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:32.5714517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:32.5718321Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162632.xml
2022-09-27T16:26:32.5721073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:32.5722003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:32.9296579Z skip: Skipped due to small world size. (4.176s)
2022-09-27T16:26:32.9296879Z 
2022-09-27T16:26:32.9297263Z ----------------------------------------------------------------------
2022-09-27T16:26:32.9297856Z Ran 1 test in 4.176s
2022-09-27T16:26:32.9298047Z 
2022-09-27T16:26:32.9298158Z OK (skipped=1)
2022-09-27T16:26:32.9298324Z 
2022-09-27T16:26:32.9298434Z Generating XML reports...
2022-09-27T16:26:32.9334335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162628.xml
2022-09-27T16:26:33.0788595Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:26:33.0811269Z 
2022-09-27T16:26:33.0811705Z Running tests...
2022-09-27T16:26:33.0812190Z ----------------------------------------------------------------------
2022-09-27T16:26:34.6081048Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:34.6081653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:34.6082676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:34.6083165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:34.7938832Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:34.8131168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78174
2022-09-27T16:26:34.8137985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78175
2022-09-27T16:26:35.0803787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:35.0804782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:35.0806004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:35.0806913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:35.1120055Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:35.1144086Z 
2022-09-27T16:26:35.1144488Z Running tests...
2022-09-27T16:26:35.1144942Z ----------------------------------------------------------------------
2022-09-27T16:26:35.1161875Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:35.1163475Z 
2022-09-27T16:26:35.1163832Z ----------------------------------------------------------------------
2022-09-27T16:26:35.1164171Z Ran 1 test in 0.002s
2022-09-27T16:26:35.1164325Z 
2022-09-27T16:26:35.1164437Z OK (skipped=1)
2022-09-27T16:26:35.1164596Z 
2022-09-27T16:26:35.1164721Z Generating XML reports...
2022-09-27T16:26:35.1198823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162635.xml
2022-09-27T16:26:35.6452542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:35.6474581Z 
2022-09-27T16:26:35.6474781Z Running tests...
2022-09-27T16:26:35.6475495Z ----------------------------------------------------------------------
2022-09-27T16:26:36.4965667Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:36.4966374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:36.4967265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:36.4967759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:36.5820671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:36.5821155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:36.5824192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:36.5824717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:37.0084182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:37.1015093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:37.1015946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:37.1097516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:37.1104312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:37.1644744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:37.1645260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:37.1647546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:37.1648348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:37.2028676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:37.2059713Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:37.2264291Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78278
2022-09-27T16:26:37.2269520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78279
2022-09-27T16:26:37.6216680Z skip: Skipped due to small world size. (4.540s)
2022-09-27T16:26:37.6217220Z 
2022-09-27T16:26:37.6217915Z ----------------------------------------------------------------------
2022-09-27T16:26:37.6218288Z Ran 1 test in 4.540s
2022-09-27T16:26:37.6218457Z 
2022-09-27T16:26:37.6218587Z OK (skipped=1)
2022-09-27T16:26:37.6218747Z 
2022-09-27T16:26:37.6218878Z Generating XML reports...
2022-09-27T16:26:37.6275188Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162633.xml
2022-09-27T16:26:37.7085744Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:37.7109628Z 
2022-09-27T16:26:37.7109964Z Running tests...
2022-09-27T16:26:37.7110934Z ----------------------------------------------------------------------
2022-09-27T16:26:37.7131278Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:37.7131631Z 
2022-09-27T16:26:37.7131922Z ----------------------------------------------------------------------
2022-09-27T16:26:37.7132256Z Ran 1 test in 0.002s
2022-09-27T16:26:37.7132424Z 
2022-09-27T16:26:37.7132536Z OK (skipped=1)
2022-09-27T16:26:37.7132696Z 
2022-09-27T16:26:37.7132823Z Generating XML reports...
2022-09-27T16:26:37.7168613Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162637.xml
2022-09-27T16:26:38.8940159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:38.8941318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:38.8946107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:38.8948101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:39.0388892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:39.0389629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:39.0391503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:39.0392017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:39.5692758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:39.5693355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:39.5694162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:39.5694853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:39.5700471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:39.5701216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:39.5797716Z STAGE:2022-09-27 16:26:39 78278:78278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:39.5798804Z STAGE:2022-09-27 16:26:39 78279:78279 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:39.5814880Z STAGE:2022-09-27 16:26:39 78278:78278 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:39.5815500Z STAGE:2022-09-27 16:26:39 78279:78279 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:39.5839580Z STAGE:2022-09-27 16:26:39 78278:78278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:39.5840382Z STAGE:2022-09-27 16:26:39 78279:78279 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:39.5853967Z STAGE:2022-09-27 16:26:39 78278:78278 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:26:39 78279:78279 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:39.5854376Z 
2022-09-27T16:26:39.7917511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:39.7918047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:39.7919276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:39.7919770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:39.8518516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:39.8519014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:39.8520959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:39.8521451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:39.9338820Z ok (4.286s)
2022-09-27T16:26:39.9339035Z 
2022-09-27T16:26:39.9339446Z ----------------------------------------------------------------------
2022-09-27T16:26:39.9339788Z Ran 1 test in 4.286s
2022-09-27T16:26:39.9339958Z 
2022-09-27T16:26:39.9340034Z OK
2022-09-27T16:26:39.9340179Z 
2022-09-27T16:26:39.9340317Z Generating XML reports...
2022-09-27T16:26:39.9376724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162635.xml
2022-09-27T16:26:40.3033391Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:40.3056809Z 
2022-09-27T16:26:40.3057332Z Running tests...
2022-09-27T16:26:40.3057849Z ----------------------------------------------------------------------
2022-09-27T16:26:40.3076750Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:40.3077215Z 
2022-09-27T16:26:40.3077512Z ----------------------------------------------------------------------
2022-09-27T16:26:40.3078110Z Ran 1 test in 0.002s
2022-09-27T16:26:40.3078298Z 
2022-09-27T16:26:40.3078410Z OK (skipped=1)
2022-09-27T16:26:40.3078570Z 
2022-09-27T16:26:40.3078702Z Generating XML reports...
2022-09-27T16:26:40.3112779Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162640.xml
2022-09-27T16:26:40.3518725Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:26:40.3541151Z 
2022-09-27T16:26:40.3541548Z Running tests...
2022-09-27T16:26:40.3541989Z ----------------------------------------------------------------------
2022-09-27T16:26:41.9387965Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:41.9583835Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78488
2022-09-27T16:26:41.9589337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78489
2022-09-27T16:26:42.0724270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:42.0724759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:42.0727626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:42.0728117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:42.2854142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:42.2854643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:42.2856382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:42.2856877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:42.5715634Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:42.5742191Z 
2022-09-27T16:26:42.5743019Z Running tests...
2022-09-27T16:26:42.5744120Z ----------------------------------------------------------------------
2022-09-27T16:26:42.7927897Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:42.7952758Z 
2022-09-27T16:26:42.7953096Z Running tests...
2022-09-27T16:26:42.7972060Z ----------------------------------------------------------------------
2022-09-27T16:26:42.7973105Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:42.7973730Z 
2022-09-27T16:26:42.7974269Z ----------------------------------------------------------------------
2022-09-27T16:26:42.7974906Z Ran 1 test in 0.002s
2022-09-27T16:26:42.7975230Z 
2022-09-27T16:26:42.7975414Z OK (skipped=1)
2022-09-27T16:26:42.7975715Z 
2022-09-27T16:26:42.7975984Z Generating XML reports...
2022-09-27T16:26:42.8009806Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162642.xml
2022-09-27T16:26:43.5508708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:43.5509678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:43.5511350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:43.5512261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:43.5585403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:43.5586263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:43.5588510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:43.5589373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:44.0658828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:44.0837516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:44.0838341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:44.0862194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:44.0869762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:44.1852616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:44.2544072Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:44.2754014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78592
2022-09-27T16:26:44.2758966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78593
2022-09-27T16:26:44.5654193Z skip: Skipped due to small world size. (4.211s)
2022-09-27T16:26:44.5654438Z 
2022-09-27T16:26:44.5654820Z ----------------------------------------------------------------------
2022-09-27T16:26:44.5655168Z Ran 1 test in 4.211s
2022-09-27T16:26:44.5655315Z 
2022-09-27T16:26:44.5655427Z OK (skipped=1)
2022-09-27T16:26:44.5655583Z 
2022-09-27T16:26:44.5655922Z Generating XML reports...
2022-09-27T16:26:44.5691432Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162640.xml
2022-09-27T16:26:44.7494766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:44.7495272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:44.7497328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:44.7497822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:45.2998764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:45.3021179Z 
2022-09-27T16:26:45.3021416Z Running tests...
2022-09-27T16:26:45.3021873Z ----------------------------------------------------------------------
2022-09-27T16:26:45.3041023Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:45.3041817Z 
2022-09-27T16:26:45.3042266Z ----------------------------------------------------------------------
2022-09-27T16:26:45.3042617Z Ran 1 test in 0.002s
2022-09-27T16:26:45.3042787Z 
2022-09-27T16:26:45.3042901Z OK (skipped=1)
2022-09-27T16:26:45.3043041Z 
2022-09-27T16:26:45.3043194Z Generating XML reports...
2022-09-27T16:26:45.3076718Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162645.xml
2022-09-27T16:26:45.9506431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:45.9506938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:45.9508113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:45.9508600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:45.9824535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:45.9825008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:45.9828032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:45.9828922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:46.5675966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:46.5676536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:46.5677327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:46.5678043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:46.5783727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:46.6689041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:46.7191078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:46.7191581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:46.7193830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:46.7194638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:47.0826907Z ok (4.508s)
2022-09-27T16:26:47.0827108Z 
2022-09-27T16:26:47.0827484Z ----------------------------------------------------------------------
2022-09-27T16:26:47.0827834Z Ran 1 test in 4.509s
2022-09-27T16:26:47.0828006Z 
2022-09-27T16:26:47.0828106Z OK
2022-09-27T16:26:47.0828249Z 
2022-09-27T16:26:47.0828386Z Generating XML reports...
2022-09-27T16:26:47.0863397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162642.xml
2022-09-27T16:26:47.2218305Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:26:47.2240058Z 
2022-09-27T16:26:47.2240480Z Running tests...
2022-09-27T16:26:47.2241170Z ----------------------------------------------------------------------
2022-09-27T16:26:47.2322738Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:47.2323284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:47.2327810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:47.2328301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:47.7275565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:47.7298940Z 
2022-09-27T16:26:47.7299238Z Running tests...
2022-09-27T16:26:47.7299699Z ----------------------------------------------------------------------
2022-09-27T16:26:47.7319620Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-09-27T16:26:47.7320619Z 
2022-09-27T16:26:47.7320941Z ----------------------------------------------------------------------
2022-09-27T16:26:47.7321467Z Ran 1 test in 0.002s
2022-09-27T16:26:47.7321737Z 
2022-09-27T16:26:47.7321851Z OK (skipped=1)
2022-09-27T16:26:47.7322012Z 
2022-09-27T16:26:47.7322121Z Generating XML reports...
2022-09-27T16:26:47.7356407Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162647.xml
2022-09-27T16:26:48.8990844Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:48.9446597Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78799
2022-09-27T16:26:48.9451209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78800
2022-09-27T16:26:49.2286133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:49.2286664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:49.2288609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:49.2289124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:49.7835758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:49.7836268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:49.7838614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:49.7839116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:49.8290421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:49.8314030Z 
2022-09-27T16:26:49.8314299Z Running tests...
2022-09-27T16:26:49.8314755Z ----------------------------------------------------------------------
2022-09-27T16:26:50.2846675Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:50.2868865Z 
2022-09-27T16:26:50.2869179Z Running tests...
2022-09-27T16:26:50.2869616Z ----------------------------------------------------------------------
2022-09-27T16:26:50.2892462Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-09-27T16:26:50.2892821Z 
2022-09-27T16:26:50.2893163Z ----------------------------------------------------------------------
2022-09-27T16:26:50.2893520Z Ran 1 test in 0.002s
2022-09-27T16:26:50.2893690Z 
2022-09-27T16:26:50.2893782Z OK (skipped=1)
2022-09-27T16:26:50.2893940Z 
2022-09-27T16:26:50.2894084Z Generating XML reports...
2022-09-27T16:26:50.2928158Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162650.xml
2022-09-27T16:26:50.5702622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:50.5703219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:50.5704717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:50.5705194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:50.6381318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:50.6381828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:50.6382778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:50.6383274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:51.0803969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:51.1650382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:51.1651239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:51.1715864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:51.1722946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:51.2713979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:51.4780891Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:51.4983835Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78903
2022-09-27T16:26:51.4988863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78904
2022-09-27T16:26:51.6516126Z skip: Skipped due to small world size. (4.427s)
2022-09-27T16:26:51.6516421Z 
2022-09-27T16:26:51.6516812Z ----------------------------------------------------------------------
2022-09-27T16:26:51.6517149Z Ran 1 test in 4.427s
2022-09-27T16:26:51.6517320Z 
2022-09-27T16:26:51.6517414Z OK (skipped=1)
2022-09-27T16:26:51.6517567Z 
2022-09-27T16:26:51.6517694Z Generating XML reports...
2022-09-27T16:26:51.6553382Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162647.xml
2022-09-27T16:26:52.3245698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:52.3246698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:52.3248364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:52.3249356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:52.8908838Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:52.8933315Z 
2022-09-27T16:26:52.8933592Z Running tests...
2022-09-27T16:26:52.8934016Z ----------------------------------------------------------------------
2022-09-27T16:26:52.8954494Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:26:52.8955097Z 
2022-09-27T16:26:52.8955597Z ----------------------------------------------------------------------
2022-09-27T16:26:52.8955970Z Ran 1 test in 0.002s
2022-09-27T16:26:52.8956138Z 
2022-09-27T16:26:52.8956234Z OK (skipped=1)
2022-09-27T16:26:52.8956395Z 
2022-09-27T16:26:52.8956523Z Generating XML reports...
2022-09-27T16:26:52.8992061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162652.xml
2022-09-27T16:26:53.1334324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:53.1334843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:53.1336211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:53.1336805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:53.1793116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:53.1793603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:53.1795728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:53.1796473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:53.7004286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:53.7004876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:53.7005656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:53.7006363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:53.7012126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:53.7013569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:53.7105222Z STAGE:2022-09-27 16:26:53 78903:78903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:53.7106354Z STAGE:2022-09-27 16:26:53 78904:78904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:53.7123447Z STAGE:2022-09-27 16:26:53 78903:78903 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:26:53 78904:78904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:53.7124212Z 
2022-09-27T16:26:53.7148221Z STAGE:2022-09-27 16:26:53 78903:78903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:53.7148876Z STAGE:2022-09-27 16:26:53 78904:78904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:53.7164551Z STAGE:2022-09-27 16:26:53 78903:78903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:53.7165765Z STAGE:2022-09-27 16:26:53 78904:78904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:26:53.7535981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:53.7536833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:53.7537858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:53.7539082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:54.1053817Z ok (4.274s)
2022-09-27T16:26:54.1054023Z 
2022-09-27T16:26:54.1054423Z ----------------------------------------------------------------------
2022-09-27T16:26:54.1054758Z Ran 1 test in 4.274s
2022-09-27T16:26:54.1054929Z 
2022-09-27T16:26:54.1055025Z OK
2022-09-27T16:26:54.1055159Z 
2022-09-27T16:26:54.1055294Z Generating XML reports...
2022-09-27T16:26:54.1090135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162649.xml
2022-09-27T16:26:54.3079467Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:26:54.3103109Z 
2022-09-27T16:26:54.3103524Z Running tests...
2022-09-27T16:26:54.3104386Z ----------------------------------------------------------------------
2022-09-27T16:26:54.8434015Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:54.8435144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:54.8436320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:54.8437191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:55.4124508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:26:55.4147108Z 
2022-09-27T16:26:55.4147573Z Running tests...
2022-09-27T16:26:55.4148324Z ----------------------------------------------------------------------
2022-09-27T16:26:55.9914957Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:56.0116548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79083
2022-09-27T16:26:56.0121972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79084
2022-09-27T16:26:56.1058003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:56.1058508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:56.1060603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:56.1061114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:56.6013534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:26:56.6035632Z 
2022-09-27T16:26:56.6036005Z Running tests...
2022-09-27T16:26:56.6036759Z ----------------------------------------------------------------------
2022-09-27T16:26:57.0851815Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:57.1051455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79150
2022-09-27T16:26:57.1057546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79151
2022-09-27T16:26:57.6536255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:57.6536767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:57.6537738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:57.6538236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:57.7169689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:57.7170799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:57.7172042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:57.7173494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:58.1699486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:58.2978946Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:26:58.3040781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:58.3041846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:58.3118460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:58.3127365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:58.3175402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79219
2022-09-27T16:26:58.3181916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79220
2022-09-27T16:26:58.3197582Z STAGE:2022-09-27 16:26:58 79084:79084 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:58.4056299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:58.4122426Z STAGE:2022-09-27 16:26:58 79083:79083 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:26:58.7275588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:58.7276090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:58.7277108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:58.7277600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:58.7299253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:58.7299705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:58.7303634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:58.7304117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:26:59.3294176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:26:59.3341561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:26:59.3342620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:59.3397171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:26:59.3403859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:26:59.4356875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:26:59.9825961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:26:59.9826464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:26:59.9827267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:26:59.9827754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:00.0097877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:00.0098380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:00.0099576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:00.0100052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:00.1515476Z [1664296020.150988] [28fd6574525b:79084:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:00.1516013Z [1664296020.150988] [28fd6574525b:79084:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:00.1617801Z [1664296020.161268] [28fd6574525b:79083:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:00.1618331Z [1664296020.161268] [28fd6574525b:79083:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:00.5601530Z STAGE:2022-09-27 16:27:00 79083:79083 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:00 79084:79084 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:00.5601994Z 
2022-09-27T16:27:00.5683692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:00.5684235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:00.5684986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:00.5685676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:00.5795595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:00.5863657Z STAGE:2022-09-27 16:27:00 79220:79220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:00.6699370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:00.6769908Z STAGE:2022-09-27 16:27:00 79219:79219 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:00.6787322Z STAGE:2022-09-27 16:27:00 79219:79219 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:00.6787930Z STAGE:2022-09-27 16:27:00 79220:79220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:00.6814473Z STAGE:2022-09-27 16:27:00 79219:79219 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:27:00 79220:79220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:00.6815082Z 
2022-09-27T16:27:00.6829244Z STAGE:2022-09-27 16:27:00 79219:79219 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:00.6830163Z STAGE:2022-09-27 16:27:00 79220:79220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:00.6981131Z STAGE:2022-09-27 16:27:00 79083:79083 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:00.6998964Z STAGE:2022-09-27 16:27:00 79084:79084 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:00.7010175Z STAGE:2022-09-27 16:27:00 79083:79083 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:00.7010768Z STAGE:2022-09-27 16:27:00 79084:79084 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:01.1256178Z ok (4.522s)
2022-09-27T16:27:01.1256408Z 
2022-09-27T16:27:01.1256767Z ----------------------------------------------------------------------
2022-09-27T16:27:01.1257106Z Ran 1 test in 4.522s
2022-09-27T16:27:01.1257273Z 
2022-09-27T16:27:01.1257367Z OK
2022-09-27T16:27:01.1257504Z 
2022-09-27T16:27:01.1257641Z Generating XML reports...
2022-09-27T16:27:01.1292963Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162656.xml
2022-09-27T16:27:01.8256174Z ok (7.515s)
2022-09-27T16:27:01.8256395Z 
2022-09-27T16:27:01.8257097Z ----------------------------------------------------------------------
2022-09-27T16:27:01.8257462Z Ran 1 test in 7.515s
2022-09-27T16:27:01.8257610Z 
2022-09-27T16:27:01.8257704Z OK
2022-09-27T16:27:01.8257840Z 
2022-09-27T16:27:01.8257973Z Generating XML reports...
2022-09-27T16:27:01.8308438Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162654.xml
2022-09-27T16:27:02.6187790Z ok (7.204s)
2022-09-27T16:27:02.6188206Z 
2022-09-27T16:27:02.6189011Z ----------------------------------------------------------------------
2022-09-27T16:27:02.6189535Z Ran 1 test in 7.204s
2022-09-27T16:27:02.6189708Z 
2022-09-27T16:27:02.6189802Z OK
2022-09-27T16:27:02.6189923Z 
2022-09-27T16:27:02.6190058Z Generating XML reports...
2022-09-27T16:27:02.6225398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162655.xml
2022-09-27T16:27:03.3463232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:03.3463770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:03.3465944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:03.3466627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:03.8275420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:27:03.8301184Z 
2022-09-27T16:27:03.8301666Z Running tests...
2022-09-27T16:27:03.8302164Z ----------------------------------------------------------------------
2022-09-27T16:27:04.1643368Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:04.1643971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:04.1646972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:04.1647476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:04.6544475Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:27:04.6566553Z 
2022-09-27T16:27:04.6566892Z Running tests...
2022-09-27T16:27:04.7597539Z ----------------------------------------------------------------------
2022-09-27T16:27:04.7598360Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:04.7599209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:04.7600308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:04.7600834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:05.2747913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:05.2772336Z 
2022-09-27T16:27:05.2772930Z Running tests...
2022-09-27T16:27:05.2773453Z ----------------------------------------------------------------------
2022-09-27T16:27:05.2792238Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:27:05.2793456Z 
2022-09-27T16:27:05.2793744Z ----------------------------------------------------------------------
2022-09-27T16:27:05.2794088Z Ran 1 test in 0.002s
2022-09-27T16:27:05.2794258Z 
2022-09-27T16:27:05.2794370Z OK (skipped=1)
2022-09-27T16:27:05.2794530Z 
2022-09-27T16:27:05.2794681Z Generating XML reports...
2022-09-27T16:27:05.2830157Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162705.xml
2022-09-27T16:27:05.4481676Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:05.4682905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79430
2022-09-27T16:27:05.4688738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79431
2022-09-27T16:27:06.2696258Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:06.2895739Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79496
2022-09-27T16:27:06.2900801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79497
2022-09-27T16:27:07.1791358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:07.1791878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:07.1793238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:07.1793730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:07.2532972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:07.2533468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:07.2535833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:07.2536331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:07.3940705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:07.3941204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:07.3942968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:07.3943460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:07.7940025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:07.7940640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:07.7941421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:07.7942114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:07.7948972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:07.7949500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:07.8048656Z STAGE:2022-09-27 16:27:07 79430:79430 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:07.8049289Z STAGE:2022-09-27 16:27:07 79431:79431 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:07.8068225Z STAGE:2022-09-27 16:27:07 79430:79430 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:07 79431:79431 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:07.8068637Z 
2022-09-27T16:27:07.8098635Z STAGE:2022-09-27 16:27:07 79431:79431 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:27:07 79430:79430 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:07.8099033Z 
2022-09-27T16:27:07.8114878Z STAGE:2022-09-27 16:27:07 79430:79430 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:07 79431:79431 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:07.8115279Z 
2022-09-27T16:27:07.9040534Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:07.9062350Z 
2022-09-27T16:27:07.9062784Z Running tests...
2022-09-27T16:27:07.9063312Z ----------------------------------------------------------------------
2022-09-27T16:27:07.9083219Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:27:07.9083556Z 
2022-09-27T16:27:07.9083854Z ----------------------------------------------------------------------
2022-09-27T16:27:07.9084191Z Ran 1 test in 0.002s
2022-09-27T16:27:07.9084358Z 
2022-09-27T16:27:07.9084477Z OK (skipped=1)
2022-09-27T16:27:07.9084634Z 
2022-09-27T16:27:07.9084742Z Generating XML reports...
2022-09-27T16:27:07.9119897Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162707.xml
2022-09-27T16:27:07.9348270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:07.9348789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:07.9349943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:07.9350445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:07.9620444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:07.9621210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:07.9623019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:07.9623509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:08.2760321Z ok (4.446s)
2022-09-27T16:27:08.2760812Z 
2022-09-27T16:27:08.2761186Z ----------------------------------------------------------------------
2022-09-27T16:27:08.2761547Z Ran 1 test in 4.446s
2022-09-27T16:27:08.2761716Z 
2022-09-27T16:27:08.2761809Z OK
2022-09-27T16:27:08.2761931Z 
2022-09-27T16:27:08.2762067Z Generating XML reports...
2022-09-27T16:27:08.2797630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162703.xml
2022-09-27T16:27:08.4638193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:08.4914264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:08.4915054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:08.4942493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:08.4950561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:08.5017418Z STAGE:2022-09-27 16:27:08 79497:79497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:08.5927775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:08.5992876Z STAGE:2022-09-27 16:27:08 79496:79496 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:09.9328308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:09.9329025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:09.9329895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:09.9330712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:10.3131421Z [1664296030.312638] [28fd6574525b:79496:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:10.3131988Z [1664296030.312638] [28fd6574525b:79496:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:10.3133043Z [1664296030.312975] [28fd6574525b:79497:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:10.3133513Z [1664296030.312975] [28fd6574525b:79497:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:10.4109399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:10.4109890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:10.4112259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:10.4112749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:10.4540009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:10.4563846Z 
2022-09-27T16:27:10.4564331Z Running tests...
2022-09-27T16:27:10.4564845Z ----------------------------------------------------------------------
2022-09-27T16:27:10.4585215Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:27:10.4585804Z 
2022-09-27T16:27:10.4586112Z ----------------------------------------------------------------------
2022-09-27T16:27:10.4586431Z Ran 1 test in 0.002s
2022-09-27T16:27:10.4586599Z 
2022-09-27T16:27:10.4586708Z OK (skipped=1)
2022-09-27T16:27:10.4586864Z 
2022-09-27T16:27:10.4586988Z Generating XML reports...
2022-09-27T16:27:10.4621024Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162710.xml
2022-09-27T16:27:10.6995508Z STAGE:2022-09-27 16:27:10 79497:79497 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:10 79496:79496 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:10.6995954Z 
2022-09-27T16:27:10.8248320Z STAGE:2022-09-27 16:27:10 79497:79497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:10.8248991Z STAGE:2022-09-27 16:27:10 79496:79496 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:10.8257665Z STAGE:2022-09-27 16:27:10 79496:79496 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:10 79497:79497 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:10.8258089Z 
2022-09-27T16:27:10.9222347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:27:10.9245423Z 
2022-09-27T16:27:10.9245685Z Running tests...
2022-09-27T16:27:10.9246124Z ----------------------------------------------------------------------
2022-09-27T16:27:12.1031382Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... ok (7.446s)
2022-09-27T16:27:12.1031779Z 
2022-09-27T16:27:12.1032155Z ----------------------------------------------------------------------
2022-09-27T16:27:12.1032508Z Ran 1 test in 7.446s
2022-09-27T16:27:12.1032674Z 
2022-09-27T16:27:12.1032769Z OK
2022-09-27T16:27:12.1032887Z 
2022-09-27T16:27:12.1033016Z Generating XML reports...
2022-09-27T16:27:12.1067870Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162704.xml
2022-09-27T16:27:12.4567923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:12.4568454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:12.4570907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:12.4571405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:12.6449925Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:12.6650262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79725
2022-09-27T16:27:12.6656718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79726
2022-09-27T16:27:12.9574400Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:12.9597502Z 
2022-09-27T16:27:12.9597855Z Running tests...
2022-09-27T16:27:12.9598289Z ----------------------------------------------------------------------
2022-09-27T16:27:14.2974263Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:14.2974871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:14.2976943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:14.2977457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:14.3306105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:14.3306624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:14.3309025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:14.3309544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:14.3762837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:14.3763379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:14.3765672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:14.3766170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:14.7084121Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:14.7284772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79825
2022-09-27T16:27:14.7291530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79826
2022-09-27T16:27:14.8299029Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:27:14.8321801Z 
2022-09-27T16:27:14.8336942Z Running tests...
2022-09-27T16:27:14.8337887Z ----------------------------------------------------------------------
2022-09-27T16:27:14.9018663Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:14.9019825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:14.9021508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:14.9022862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:14.9026345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:14.9027239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:15.2736196Z ok (4.349s)
2022-09-27T16:27:15.2736615Z 
2022-09-27T16:27:15.2737241Z ----------------------------------------------------------------------
2022-09-27T16:27:15.2737870Z Ran 1 test in 4.349s
2022-09-27T16:27:15.2738202Z 
2022-09-27T16:27:15.2738382Z OK
2022-09-27T16:27:15.2738647Z 
2022-09-27T16:27:15.2738858Z Generating XML reports...
2022-09-27T16:27:15.2774980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162710.xml
2022-09-27T16:27:16.3696014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:16.3697148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:16.3699118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:16.3700293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:16.4123975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:16.4125114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:16.4131189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:16.4132367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:16.5483625Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:16.5682285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79933
2022-09-27T16:27:16.5689651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79934
2022-09-27T16:27:16.9402920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:16.9630289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:16.9631144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:16.9708674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:16.9717039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:17.0644788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:17.4142810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:17.4143342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:17.4144542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:17.4145339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:17.9273161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:27:17.9297344Z 
2022-09-27T16:27:17.9297584Z Running tests...
2022-09-27T16:27:17.9298042Z ----------------------------------------------------------------------
2022-09-27T16:27:18.2164355Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:18.2165308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:18.2165967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:18.2166472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:18.2653851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:18.2654331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:18.2655564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:18.2656050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:18.6915060Z STAGE:2022-09-27 16:27:18 79825:79825 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:18.7473933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:18.7831715Z STAGE:2022-09-27 16:27:18 79826:79826 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:18.8057814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:18.8058583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:18.8081414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:18.8088957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:18.8680912Z STAGE:2022-09-27 16:27:18 79826:79826 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:18.8681601Z STAGE:2022-09-27 16:27:18 79825:79825 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:18.9072913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:19.0411306Z STAGE:2022-09-27 16:27:19 79826:79826 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:19.0414873Z STAGE:2022-09-27 16:27:19 79825:79825 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:19.0430951Z STAGE:2022-09-27 16:27:19 79826:79826 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:19.0432355Z STAGE:2022-09-27 16:27:19 79825:79825 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:19.0455098Z STAGE:2022-09-27 16:27:19 79826:79826 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:19.0456180Z STAGE:2022-09-27 16:27:19 79825:79825 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:19.0468590Z STAGE:2022-09-27 16:27:19 79826:79826 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:19.0469217Z STAGE:2022-09-27 16:27:19 79825:79825 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:19.5793917Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:19.5994434Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80025
2022-09-27T16:27:19.6000582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80026
2022-09-27T16:27:19.6403792Z ok (6.680s)
2022-09-27T16:27:19.6404007Z 
2022-09-27T16:27:19.6404400Z ----------------------------------------------------------------------
2022-09-27T16:27:19.6404723Z Ran 1 test in 6.681s
2022-09-27T16:27:19.6404896Z 
2022-09-27T16:27:19.6404992Z OK
2022-09-27T16:27:19.6405131Z 
2022-09-27T16:27:19.6405266Z Generating XML reports...
2022-09-27T16:27:19.6459048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162712.xml
2022-09-27T16:27:20.6019614Z STAGE:2022-09-27 16:27:20 79933:79933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:20.6414862Z STAGE:2022-09-27 16:27:20 79934:79934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:20.6565100Z [1664296040.656157] [28fd6574525b:79933:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:20.6565631Z [1664296040.656157] [28fd6574525b:79933:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:20.6650592Z [1664296040.664739] [28fd6574525b:79934:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:20.6651103Z [1664296040.664739] [28fd6574525b:79934:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:21.1429082Z STAGE:2022-09-27 16:27:21 79933:79933 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:21 79934:79934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:21.1429566Z 
2022-09-27T16:27:21.2681047Z STAGE:2022-09-27 16:27:21 79934:79934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:21.2685050Z STAGE:2022-09-27 16:27:21 79933:79933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:21.2695400Z STAGE:2022-09-27 16:27:21 79934:79934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:21.2700193Z STAGE:2022-09-27 16:27:21 79933:79933 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:21.4005304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:21.4005811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:21.4008145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:21.4008667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:21.4184313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:21.4184801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:21.4187003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:21.4187518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:21.9355465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:21.9356258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:21.9357075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:21.9357567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:21.9495945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:21.9496486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:21.9497188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:21.9497896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:21.9606107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:21.9819188Z ok (7.150s)
2022-09-27T16:27:21.9819503Z 
2022-09-27T16:27:21.9819989Z ----------------------------------------------------------------------
2022-09-27T16:27:21.9820377Z Ran 1 test in 7.150s
2022-09-27T16:27:21.9820558Z 
2022-09-27T16:27:21.9820657Z OK
2022-09-27T16:27:21.9820796Z 
2022-09-27T16:27:21.9820913Z Generating XML reports...
2022-09-27T16:27:21.9857584Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162714.xml
2022-09-27T16:27:22.0510837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:22.0719768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:27:22.0720646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:27:22.0721529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:27:22.0722332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:27:22.0816137Z STAGE:2022-09-27 16:27:22 80026:80026 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:22.0816751Z STAGE:2022-09-27 16:27:22 80025:80025 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:22.0830379Z STAGE:2022-09-27 16:27:22 80025:80025 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:22 80026:80026 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:22.0831128Z 
2022-09-27T16:27:22.0847679Z STAGE:2022-09-27 16:27:22 80025:80025 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:27:22 80026:80026 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:22.0848069Z 
2022-09-27T16:27:22.0861562Z STAGE:2022-09-27 16:27:22 80026:80026 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:22 80025:80025 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:22.0861963Z 
2022-09-27T16:27:22.4392340Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:22.4415469Z 
2022-09-27T16:27:22.4415892Z Running tests...
2022-09-27T16:27:22.4416399Z ----------------------------------------------------------------------
2022-09-27T16:27:22.6073337Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... ok (4.677s)
2022-09-27T16:27:22.6073685Z 
2022-09-27T16:27:22.6074057Z ----------------------------------------------------------------------
2022-09-27T16:27:22.6074400Z Ran 1 test in 4.677s
2022-09-27T16:27:22.6074566Z 
2022-09-27T16:27:22.6074658Z OK
2022-09-27T16:27:22.6074777Z 
2022-09-27T16:27:22.6074904Z Generating XML reports...
2022-09-27T16:27:22.6108912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162717.xml
2022-09-27T16:27:24.1461202Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:24.1628869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:24.1629360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:24.1633248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:24.1633761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:24.1657298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80221
2022-09-27T16:27:24.1663846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80222
2022-09-27T16:27:24.6465103Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:27:24.6487700Z 
2022-09-27T16:27:24.6487984Z Running tests...
2022-09-27T16:27:24.6488427Z ----------------------------------------------------------------------
2022-09-27T16:27:24.7421859Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:24.7423145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:24.7424809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:24.7425312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:25.3089260Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:27:25.3112997Z 
2022-09-27T16:27:25.3113429Z Running tests...
2022-09-27T16:27:25.3113915Z ----------------------------------------------------------------------
2022-09-27T16:27:25.7899048Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:25.7899665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:25.7900650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:25.7901262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:25.8018308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:25.8019106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:25.8021521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:25.8022016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:26.2894544Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:26.3095673Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80292
2022-09-27T16:27:26.3096201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:26.3100844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80295
2022-09-27T16:27:26.3258457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:26.3259279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:26.3300127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:26.3306812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:26.4273678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:26.9588627Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:27.0051449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80365
2022-09-27T16:27:27.0146521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80366
2022-09-27T16:27:28.0129149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:28.0129976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:28.0130972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:28.0131472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:28.0178137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:28.0178611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:28.0181639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:28.0182137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:28.0579565Z STAGE:2022-09-27 16:27:28 80221:80221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:28.1120973Z STAGE:2022-09-27 16:27:28 80222:80222 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:28.1826435Z STAGE:2022-09-27 16:27:28 80222:80222 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:28.1827115Z STAGE:2022-09-27 16:27:28 80221:80221 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:28.3498450Z STAGE:2022-09-27 16:27:28 80222:80222 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:28.3520211Z STAGE:2022-09-27 16:27:28 80221:80221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:28.3536139Z STAGE:2022-09-27 16:27:28 80222:80222 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:28.3537209Z STAGE:2022-09-27 16:27:28 80221:80221 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:28.3558444Z STAGE:2022-09-27 16:27:28 80222:80222 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:28.3559298Z STAGE:2022-09-27 16:27:28 80221:80221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:28.3572642Z STAGE:2022-09-27 16:27:28 80222:80222 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:28.3573577Z STAGE:2022-09-27 16:27:28 80221:80221 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:28.5412388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:28.5412965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:28.5413745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:28.5414457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:28.5421729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:28.5422242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:28.7334764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:28.7335323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:28.7336507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:28.7337003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:28.7980668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:28.7981191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:28.7982785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:28.7983277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:28.8774520Z ok (6.436s)
2022-09-27T16:27:28.8774787Z 
2022-09-27T16:27:28.8775165Z ----------------------------------------------------------------------
2022-09-27T16:27:28.8775508Z Ran 1 test in 6.436s
2022-09-27T16:27:28.8775678Z 
2022-09-27T16:27:28.8775778Z OK
2022-09-27T16:27:28.8775896Z 
2022-09-27T16:27:28.8776033Z Generating XML reports...
2022-09-27T16:27:28.8811171Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162722.xml
2022-09-27T16:27:29.3938336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:29.3938926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:29.3939984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:29.3940732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:29.4047563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:29.4954308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:29.5063922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:27:29.5064661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:27:29.5065837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:27:29.5066560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:27:29.5167469Z STAGE:2022-09-27 16:27:29 80366:80366 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:29.5168089Z STAGE:2022-09-27 16:27:29 80365:80365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:29.5183078Z STAGE:2022-09-27 16:27:29 80366:80366 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:29 80365:80365 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:29.5183497Z 
2022-09-27T16:27:29.5201096Z STAGE:2022-09-27 16:27:29 80366:80366 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:27:29 80365:80365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:29.5201645Z 
2022-09-27T16:27:29.5215316Z STAGE:2022-09-27 16:27:29 80366:80366 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:29.5215955Z STAGE:2022-09-27 16:27:29 80365:80365 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:29.9217807Z ok (4.610s)
2022-09-27T16:27:29.9218038Z 
2022-09-27T16:27:29.9218451Z ----------------------------------------------------------------------
2022-09-27T16:27:29.9218833Z Ran 1 test in 4.610s
2022-09-27T16:27:29.9219008Z 
2022-09-27T16:27:29.9219104Z OK
2022-09-27T16:27:29.9219224Z 
2022-09-27T16:27:29.9219362Z Generating XML reports...
2022-09-27T16:27:29.9257058Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162725.xml
2022-09-27T16:27:30.2399122Z STAGE:2022-09-27 16:27:30 80295:80295 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:30.3695962Z STAGE:2022-09-27 16:27:30 80292:80292 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:30.3841547Z [1664296050.383735] [28fd6574525b:80292:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:30.3842098Z [1664296050.383735] [28fd6574525b:80292:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:30.4108524Z [1664296050.410318] [28fd6574525b:80295:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:30.4109077Z [1664296050.410318] [28fd6574525b:80295:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:30.8293977Z STAGE:2022-09-27 16:27:30 80295:80295 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:30.8296157Z STAGE:2022-09-27 16:27:30 80292:80292 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:30.9540738Z STAGE:2022-09-27 16:27:30 80295:80295 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:30.9542122Z STAGE:2022-09-27 16:27:30 80292:80292 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:30.9555632Z STAGE:2022-09-27 16:27:30 80295:80295 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:30.9557816Z STAGE:2022-09-27 16:27:30 80292:80292 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:31.1308735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:31.1309968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:31.1314439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:31.1315391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:31.6565971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:31.6588965Z 
2022-09-27T16:27:31.6589107Z Running tests...
2022-09-27T16:27:31.6589960Z ----------------------------------------------------------------------
2022-09-27T16:27:31.7220573Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... ok (7.073s)
2022-09-27T16:27:31.7220893Z 
2022-09-27T16:27:31.7221301Z ----------------------------------------------------------------------
2022-09-27T16:27:31.7222018Z Ran 1 test in 7.073s
2022-09-27T16:27:31.7222385Z 
2022-09-27T16:27:31.7222484Z OK
2022-09-27T16:27:31.7222630Z 
2022-09-27T16:27:31.7222762Z Generating XML reports...
2022-09-27T16:27:31.7257456Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162724.xml
2022-09-27T16:27:32.0760622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:32.0761198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:32.0762389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:32.0762882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:32.5927915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:27:32.5949935Z 
2022-09-27T16:27:32.5950180Z Running tests...
2022-09-27T16:27:32.5950638Z ----------------------------------------------------------------------
2022-09-27T16:27:33.3531080Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:33.3732795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80580
2022-09-27T16:27:33.3738401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80581
2022-09-27T16:27:33.7833934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:33.7834429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:33.7837276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:33.7837774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:34.2197535Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:34.2392662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80646
2022-09-27T16:27:34.2398380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80647
2022-09-27T16:27:34.3391318Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:27:34.3414221Z 
2022-09-27T16:27:34.3414666Z Running tests...
2022-09-27T16:27:34.3415174Z ----------------------------------------------------------------------
2022-09-27T16:27:35.0074658Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:35.0075855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:35.0077776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:35.0078782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:35.0901386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:35.0902376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:35.0903538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:35.0904492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:35.5275279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:35.6186370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:35.6187924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:35.6189442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:35.6194671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:35.6195694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:35.8616048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:35.8617063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:35.8618273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:35.8619252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:35.8899964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:35.8900895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:35.8902056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:35.8902980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:36.0198198Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:36.0403759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80724
2022-09-27T16:27:36.0409301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80725
2022-09-27T16:27:36.4430389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:36.4431509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:36.4433048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:36.4434467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:36.4439354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:36.4440670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:36.4651141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:27:36.4652150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:27:36.4653814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:27:36.4655540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:27:36.4751294Z STAGE:2022-09-27 16:27:36 80647:80647 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:27:36 80646:80646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:36.4752074Z 
2022-09-27T16:27:36.4766360Z STAGE:2022-09-27 16:27:36 80647:80647 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:36 80646:80646 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:36.4767180Z 
2022-09-27T16:27:36.4782710Z STAGE:2022-09-27 16:27:36 80647:80647 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:36.4783946Z STAGE:2022-09-27 16:27:36 80646:80646 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:36.4798580Z STAGE:2022-09-27 16:27:36 80647:80647 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:36.4799859Z STAGE:2022-09-27 16:27:36 80646:80646 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:36.9478394Z ok (4.352s)
2022-09-27T16:27:36.9478764Z 
2022-09-27T16:27:36.9479398Z ----------------------------------------------------------------------
2022-09-27T16:27:36.9480853Z Ran 1 test in 4.353s
2022-09-27T16:27:36.9481136Z 
2022-09-27T16:27:36.9481310Z OK
2022-09-27T16:27:36.9481553Z 
2022-09-27T16:27:36.9481786Z Generating XML reports...
2022-09-27T16:27:36.9536152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162732.xml
2022-09-27T16:27:37.3222919Z STAGE:2022-09-27 16:27:37 80580:80580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:37.3330107Z STAGE:2022-09-27 16:27:37 80581:80581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:37.4143541Z STAGE:2022-09-27 16:27:37 80580:80580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:37.4144721Z STAGE:2022-09-27 16:27:37 80581:80581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:37.5922418Z STAGE:2022-09-27 16:27:37 80581:80581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:37.6067745Z STAGE:2022-09-27 16:27:37 80580:80580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:37.6083473Z STAGE:2022-09-27 16:27:37 80581:80581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:37.6085014Z STAGE:2022-09-27 16:27:37 80580:80580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:37.6108348Z STAGE:2022-09-27 16:27:37 80580:80580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:37.6109571Z STAGE:2022-09-27 16:27:37 80581:80581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:37.6120668Z STAGE:2022-09-27 16:27:37 80580:80580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:37.6124584Z STAGE:2022-09-27 16:27:37 80581:80581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:37.7093011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:37.7093959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:37.7095122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:37.7095997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:37.7505650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:37.7506565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:37.7507652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:37.7508893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:38.1861567Z ok (6.526s)
2022-09-27T16:27:38.1862007Z 
2022-09-27T16:27:38.1862839Z ----------------------------------------------------------------------
2022-09-27T16:27:38.1869820Z Ran 1 test in 6.527s
2022-09-27T16:27:38.1872865Z 
2022-09-27T16:27:38.1875675Z OK
2022-09-27T16:27:38.1878328Z 
2022-09-27T16:27:38.1881340Z Generating XML reports...
2022-09-27T16:27:38.1929590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162731.xml
2022-09-27T16:27:38.2864849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:38.3397677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:38.3398479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:38.3473532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:38.3480400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:38.3547164Z STAGE:2022-09-27 16:27:38 80725:80725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:38.4411004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:38.4476454Z STAGE:2022-09-27 16:27:38 80724:80724 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:39.1926456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:39.1926967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:39.1928448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:39.1928963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:39.7705966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:27:39.7729897Z 
2022-09-27T16:27:39.7730119Z Running tests...
2022-09-27T16:27:39.7730558Z ----------------------------------------------------------------------
2022-09-27T16:27:40.1663362Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... [1664296060.165858] [28fd6574525b:80725:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:40.1663995Z [1664296060.165858] [28fd6574525b:80725:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:40.2030044Z [1664296060.202536] [28fd6574525b:80724:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:40.2030566Z [1664296060.202536] [28fd6574525b:80724:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:40.2540867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:40.2541386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:40.2543392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:40.2543889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:40.5902161Z STAGE:2022-09-27 16:27:40 80724:80724 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:40 80725:80725 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:40.5902604Z 
2022-09-27T16:27:40.7154004Z STAGE:2022-09-27 16:27:40 80725:80725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:40.7192388Z STAGE:2022-09-27 16:27:40 80724:80724 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:40.7202584Z STAGE:2022-09-27 16:27:40 80725:80725 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:40.7203185Z STAGE:2022-09-27 16:27:40 80724:80724 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:40.7572108Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:40.7597694Z 
2022-09-27T16:27:40.7597940Z Running tests...
2022-09-27T16:27:40.7598378Z ----------------------------------------------------------------------
2022-09-27T16:27:40.7618754Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:27:40.7619060Z 
2022-09-27T16:27:40.7619743Z ----------------------------------------------------------------------
2022-09-27T16:27:40.7620158Z Ran 1 test in 0.002s
2022-09-27T16:27:40.7620332Z 
2022-09-27T16:27:40.7620454Z OK (skipped=1)
2022-09-27T16:27:40.7620625Z 
2022-09-27T16:27:40.7620779Z Generating XML reports...
2022-09-27T16:27:40.7657096Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162740.xml
2022-09-27T16:27:41.3945319Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:41.4141596Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80903
2022-09-27T16:27:41.4147621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80904
2022-09-27T16:27:41.8559206Z ok (7.513s)
2022-09-27T16:27:41.8560055Z 
2022-09-27T16:27:41.8563882Z ----------------------------------------------------------------------
2022-09-27T16:27:41.8565264Z Ran 1 test in 7.514s
2022-09-27T16:27:41.8565571Z 
2022-09-27T16:27:41.8566149Z OK
2022-09-27T16:27:41.8566474Z 
2022-09-27T16:27:41.8566724Z Generating XML reports...
2022-09-27T16:27:41.8616011Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162734.xml
2022-09-27T16:27:42.8596333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:42.8596835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:42.8598743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:42.8599241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:43.0526405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:43.0527557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:43.0531773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:43.0532942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:43.1652482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:43.1652999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:43.1656039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:43.1656532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:43.3477228Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:43.3500671Z 
2022-09-27T16:27:43.3500891Z Running tests...
2022-09-27T16:27:43.3501337Z ----------------------------------------------------------------------
2022-09-27T16:27:43.3519264Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:27:43.3520994Z 
2022-09-27T16:27:43.3521462Z ----------------------------------------------------------------------
2022-09-27T16:27:43.3522097Z Ran 1 test in 0.002s
2022-09-27T16:27:43.3522288Z 
2022-09-27T16:27:43.3522402Z OK (skipped=1)
2022-09-27T16:27:43.3522561Z 
2022-09-27T16:27:43.3522670Z Generating XML reports...
2022-09-27T16:27:43.3557120Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162743.xml
2022-09-27T16:27:43.7029786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:43.7030354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:43.7031159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:43.7031863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:43.7138031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:43.8043363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:43.8152184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:27:43.8153065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:27:43.8153957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:27:43.8154642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:27:43.8249536Z STAGE:2022-09-27 16:27:43 80904:80904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:43.8250124Z STAGE:2022-09-27 16:27:43 80903:80903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:43.8264398Z STAGE:2022-09-27 16:27:43 80904:80904 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:43 80903:80903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:43.8264805Z 
2022-09-27T16:27:43.8281543Z STAGE:2022-09-27 16:27:43 80903:80903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:43.8282580Z STAGE:2022-09-27 16:27:43 80904:80904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:43.8296675Z STAGE:2022-09-27 16:27:43 80904:80904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:43.8297276Z STAGE:2022-09-27 16:27:43 80903:80903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:43.9687280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:43.9687781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:43.9689781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:43.9690280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:44.2223443Z ok (4.449s)
2022-09-27T16:27:44.2223680Z 
2022-09-27T16:27:44.2224081Z ----------------------------------------------------------------------
2022-09-27T16:27:44.2224427Z Ran 1 test in 4.449s
2022-09-27T16:27:44.2224595Z 
2022-09-27T16:27:44.2224672Z OK
2022-09-27T16:27:44.2224809Z 
2022-09-27T16:27:44.2224946Z Generating XML reports...
2022-09-27T16:27:44.2260958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162739.xml
2022-09-27T16:27:44.5388853Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:27:44.5411860Z 
2022-09-27T16:27:44.5412408Z Running tests...
2022-09-27T16:27:44.5412935Z ----------------------------------------------------------------------
2022-09-27T16:27:44.5441626Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T16:27:44.5442004Z 
2022-09-27T16:27:44.5442286Z ----------------------------------------------------------------------
2022-09-27T16:27:44.5442620Z Ran 1 test in 0.003s
2022-09-27T16:27:44.5442787Z 
2022-09-27T16:27:44.5442904Z OK (skipped=1)
2022-09-27T16:27:44.5443063Z 
2022-09-27T16:27:44.5443190Z Generating XML reports...
2022-09-27T16:27:44.5477575Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162744.xml
2022-09-27T16:27:45.3083388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:45.3086691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:45.3090159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:45.3093269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:45.9096554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:45.9118579Z 
2022-09-27T16:27:45.9118822Z Running tests...
2022-09-27T16:27:45.9119708Z ----------------------------------------------------------------------
2022-09-27T16:27:46.3636916Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:46.3637483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:46.3640864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:46.3641420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:46.6066546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:46.6067115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:46.6069624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:46.6070132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:46.8612906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:27:46.8641229Z 
2022-09-27T16:27:46.8641486Z Running tests...
2022-09-27T16:27:46.8642205Z ----------------------------------------------------------------------
2022-09-27T16:27:47.1128287Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:27:47.1150644Z 
2022-09-27T16:27:47.1150973Z Running tests...
2022-09-27T16:27:47.1151431Z ----------------------------------------------------------------------
2022-09-27T16:27:47.5109531Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:47.5312647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81155
2022-09-27T16:27:47.5318106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81156
2022-09-27T16:27:48.5109973Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:48.5304565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81220
2022-09-27T16:27:48.5310862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81221
2022-09-27T16:27:48.7371144Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:48.7566228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81223
2022-09-27T16:27:48.7573044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81224
2022-09-27T16:27:49.1733727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:49.1734263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:49.1736506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:49.1737015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:49.1998327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:49.1999913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:49.2002308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:49.2003832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:49.8063598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:49.8127119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:49.8128168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:49.8164706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:49.8171131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:49.9143182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:50.2395047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:50.2395700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:50.2396667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:50.2397416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:50.3019785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:50.3020325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:50.3021374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:50.3021959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:50.4714896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:50.4716735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:50.4717356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:50.4717872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:50.4812147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:50.4812686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:50.4814998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:50.4815506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:50.8698426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:50.8698992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:50.8699794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:50.8700731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:50.8706951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:50.8707466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:51.0044264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:51.0151407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:51.0152239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:51.0250316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:51.0256448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:51.0332174Z STAGE:2022-09-27 16:27:51 81224:81224 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:51.1165492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:51.1460300Z STAGE:2022-09-27 16:27:51 81223:81223 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:51.3380587Z skip: Skipped due to small world size. (4.474s)
2022-09-27T16:27:51.3380875Z 
2022-09-27T16:27:51.3381360Z ----------------------------------------------------------------------
2022-09-27T16:27:51.3381870Z Ran 1 test in 4.474s
2022-09-27T16:27:51.3382039Z 
2022-09-27T16:27:51.3382151Z OK (skipped=1)
2022-09-27T16:27:51.3382311Z 
2022-09-27T16:27:51.3382443Z Generating XML reports...
2022-09-27T16:27:51.3419469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162746.xml
2022-09-27T16:27:52.8603482Z [1664296072.859698] [28fd6574525b:81224:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:52.8604200Z [1664296072.859698] [28fd6574525b:81224:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:52.9696429Z [1664296072.969123] [28fd6574525b:81223:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:27:52.9697285Z [1664296072.969123] [28fd6574525b:81223:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:27:53.0439949Z ok (7.132s)
2022-09-27T16:27:53.0440730Z 
2022-09-27T16:27:53.0441497Z ----------------------------------------------------------------------
2022-09-27T16:27:53.0441921Z Ran 1 test in 7.132s
2022-09-27T16:27:53.0442096Z 
2022-09-27T16:27:53.0442192Z OK
2022-09-27T16:27:53.0442329Z 
2022-09-27T16:27:53.0442446Z Generating XML reports...
2022-09-27T16:27:53.0476987Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162745.xml
2022-09-27T16:27:53.3600071Z STAGE:2022-09-27 16:27:53 81223:81223 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:53 81224:81224 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:53.3600756Z 
2022-09-27T16:27:53.4863181Z STAGE:2022-09-27 16:27:53 81223:81223 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:53.5148236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:53.5148735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:53.5152137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:53.5152658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:53.5212006Z STAGE:2022-09-27 16:27:53 81224:81224 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:27:53.5223133Z STAGE:2022-09-27 16:27:53 81223:81223 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:27:53 81224:81224 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:27:53.5223569Z 
2022-09-27T16:27:54.0324315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:27:54.0346833Z 
2022-09-27T16:27:54.0347279Z Running tests...
2022-09-27T16:27:54.0347777Z ----------------------------------------------------------------------
2022-09-27T16:27:54.5717002Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... ok (7.456s)
2022-09-27T16:27:54.5717640Z 
2022-09-27T16:27:54.5718340Z ----------------------------------------------------------------------
2022-09-27T16:27:54.5718757Z Ran 1 test in 7.457s
2022-09-27T16:27:54.5718926Z 
2022-09-27T16:27:54.5719023Z OK
2022-09-27T16:27:54.5719161Z 
2022-09-27T16:27:54.5719310Z Generating XML reports...
2022-09-27T16:27:54.5775573Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162747.xml
2022-09-27T16:27:55.2191796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:55.2192333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:55.2193912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:55.2194428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:55.7545197Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:55.7899384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81496
2022-09-27T16:27:55.7905248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81497
2022-09-27T16:27:55.8013745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:27:55.8038078Z 
2022-09-27T16:27:55.8038532Z Running tests...
2022-09-27T16:27:55.8039029Z ----------------------------------------------------------------------
2022-09-27T16:27:56.7390786Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:56.7391924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:56.7393148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:56.7394095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:57.2321592Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:27:57.2343992Z 
2022-09-27T16:27:57.2344472Z Running tests...
2022-09-27T16:27:57.2344998Z ----------------------------------------------------------------------
2022-09-27T16:27:57.3970125Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:57.3970721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:57.3971825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:57.3972320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:57.4305695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:57.4306181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:57.4308464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:57.4309248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:57.4331608Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:57.4536012Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81566
2022-09-27T16:27:57.4541120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81567
2022-09-27T16:27:57.9590277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:57.9591382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:57.9592901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:57.9594367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:57.9698399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:58.0604522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:27:58.4990835Z skip: Skipped due to small world size. (4.464s)
2022-09-27T16:27:58.4991541Z 
2022-09-27T16:27:58.4992315Z ----------------------------------------------------------------------
2022-09-27T16:27:58.4992879Z Ran 1 test in 4.464s
2022-09-27T16:27:58.4993032Z 
2022-09-27T16:27:58.4993143Z OK (skipped=1)
2022-09-27T16:27:58.4993297Z 
2022-09-27T16:27:58.4993426Z Generating XML reports...
2022-09-27T16:27:58.5028086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162754.xml
2022-09-27T16:27:58.8760941Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:27:58.9266900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81639
2022-09-27T16:27:58.9272345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81640
2022-09-27T16:27:59.1242458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:59.1243360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:59.1243987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:59.1244478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:59.1256759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:27:59.1257231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:27:59.1260882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:27:59.1261411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:27:59.7640454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:27:59.7823655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:27:59.7825010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:59.7842975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:27:59.7851038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:27:59.8837852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:00.6131548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:00.6132625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:00.6134299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:00.6135298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:00.6370250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:00.6371213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:00.6372904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:00.6373852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:00.7081305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:00.7082300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:00.7083447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:00.7084327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:01.1556096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:01.1673836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:01.1675431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:01.1759344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:01.1766113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:01.1840974Z STAGE:2022-09-27 16:28:01 81640:81640 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:01.2391473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:28:01.2414935Z 
2022-09-27T16:28:01.2415412Z Running tests...
2022-09-27T16:28:01.2416105Z ----------------------------------------------------------------------
2022-09-27T16:28:01.2686843Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:01.2752872Z STAGE:2022-09-27 16:28:01 81639:81639 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:02.9677115Z ok (7.164s)
2022-09-27T16:28:02.9677332Z 
2022-09-27T16:28:02.9677729Z ----------------------------------------------------------------------
2022-09-27T16:28:02.9678053Z Ran 1 test in 7.164s
2022-09-27T16:28:02.9678217Z 
2022-09-27T16:28:02.9678318Z OK
2022-09-27T16:28:02.9678454Z 
2022-09-27T16:28:02.9678592Z Generating XML reports...
2022-09-27T16:28:02.9713513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162755.xml
2022-09-27T16:28:03.0410993Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:03.0624044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81770
2022-09-27T16:28:03.0629659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81771
2022-09-27T16:28:03.0869940Z [1664296083.086479] [28fd6574525b:81640:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:28:03.0870467Z [1664296083.086479] [28fd6574525b:81640:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:28:03.1305803Z [1664296083.130068] [28fd6574525b:81639:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:28:03.1306290Z [1664296083.130068] [28fd6574525b:81639:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:28:03.5577737Z STAGE:2022-09-27 16:28:03 81640:81640 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:28:03 81639:81639 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:03.5578230Z 
2022-09-27T16:28:03.6796038Z STAGE:2022-09-27 16:28:03 81640:81640 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:03.6847735Z STAGE:2022-09-27 16:28:03 81639:81639 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:03.6858008Z STAGE:2022-09-27 16:28:03 81640:81640 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:28:03 81639:81639 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:03.6858414Z 
2022-09-27T16:28:04.7409310Z ok (7.506s)
2022-09-27T16:28:04.7409518Z 
2022-09-27T16:28:04.7409919Z ----------------------------------------------------------------------
2022-09-27T16:28:04.7410264Z Ran 1 test in 7.506s
2022-09-27T16:28:04.7410433Z 
2022-09-27T16:28:04.7410507Z OK
2022-09-27T16:28:04.7410664Z 
2022-09-27T16:28:04.7410800Z Generating XML reports...
2022-09-27T16:28:04.7445869Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162757.xml
2022-09-27T16:28:04.8018227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:04.8018710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:04.8021082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:04.8021577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:04.8057350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:04.8057795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:04.8061502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:04.8061986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:05.2870948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:05.2871934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:05.2873136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:05.2874090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:05.3495866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:05.3496419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:05.3497245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:05.3497960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:05.3605835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:05.4511273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:05.7695747Z skip: Skipped due to small world size. (4.528s)
2022-09-27T16:28:05.7696010Z 
2022-09-27T16:28:05.7696419Z ----------------------------------------------------------------------
2022-09-27T16:28:05.7696742Z Ran 1 test in 4.528s
2022-09-27T16:28:05.7696911Z 
2022-09-27T16:28:05.7697023Z OK (skipped=1)
2022-09-27T16:28:05.7697182Z 
2022-09-27T16:28:05.7697316Z Generating XML reports...
2022-09-27T16:28:05.7733403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162801.xml
2022-09-27T16:28:05.7867750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:05.7909647Z 
2022-09-27T16:28:05.7909948Z Running tests...
2022-09-27T16:28:05.7910945Z ----------------------------------------------------------------------
2022-09-27T16:28:05.7943888Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.003s)
2022-09-27T16:28:05.7944660Z 
2022-09-27T16:28:05.7945405Z ----------------------------------------------------------------------
2022-09-27T16:28:05.7946161Z Ran 1 test in 0.004s
2022-09-27T16:28:05.7946573Z 
2022-09-27T16:28:05.7948007Z OK (skipped=1)
2022-09-27T16:28:05.7948339Z 
2022-09-27T16:28:05.7949600Z Generating XML reports...
2022-09-27T16:28:05.8007847Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162805.xml
2022-09-27T16:28:06.8028572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:06.8032714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:06.8034001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:06.8035385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:07.3627862Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:07.3650637Z 
2022-09-27T16:28:07.3650866Z Running tests...
2022-09-27T16:28:07.3651313Z ----------------------------------------------------------------------
2022-09-27T16:28:07.8579007Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:07.8579623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:07.8580786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:07.8581289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:07.8856633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:07.8857143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:07.8859107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:07.8859601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:08.3753881Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:08.3778274Z 
2022-09-27T16:28:08.3778634Z Running tests...
2022-09-27T16:28:08.3779125Z ----------------------------------------------------------------------
2022-09-27T16:28:08.3869428Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:28:08.3894620Z 
2022-09-27T16:28:08.3894914Z Running tests...
2022-09-27T16:28:08.3895354Z ----------------------------------------------------------------------
2022-09-27T16:28:08.9661772Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:08.9863494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81985
2022-09-27T16:28:08.9868890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81986
2022-09-27T16:28:10.0757200Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:10.0843917Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:10.0953260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82050
2022-09-27T16:28:10.0960290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82051
2022-09-27T16:28:10.1037247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82053
2022-09-27T16:28:10.1044409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82054
2022-09-27T16:28:10.6305622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:10.6306151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:10.6308710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:10.6309190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:10.6414328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:10.6414807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:10.6418221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:10.6418697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:11.3002279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:11.3807807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:11.3808713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:11.3812426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:11.3819817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:11.3888289Z STAGE:2022-09-27 16:28:11 81986:81986 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:11.4820238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:11.4887697Z STAGE:2022-09-27 16:28:11 81985:81985 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:11.7889898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:11.7890421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:11.7891739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:11.7892225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:11.8024213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:11.8024708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:11.8027996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:11.8028488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:11.8143634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:11.8144114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:11.8146811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:11.8147300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:11.9577147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:11.9577645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:11.9580140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:11.9580662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:12.3435933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:12.3703765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:12.3704628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:12.3740434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:12.3747280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:12.3750697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:28:12.4716315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:12.4720188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:28:12.4721512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:28:12.4766662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:28:12.5357583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:12.5358174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:12.5358962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:12.5359681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:12.5468381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:12.6372763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:13.1130589Z skip: Skipped due to small world size. (4.723s)
2022-09-27T16:28:13.1130847Z 
2022-09-27T16:28:13.1131230Z ----------------------------------------------------------------------
2022-09-27T16:28:13.1131568Z Ran 1 test in 4.723s
2022-09-27T16:28:13.1131735Z 
2022-09-27T16:28:13.1131849Z OK (skipped=1)
2022-09-27T16:28:13.1132007Z 
2022-09-27T16:28:13.1132137Z Generating XML reports...
2022-09-27T16:28:13.1189077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162808.xml
2022-09-27T16:28:13.1894493Z [1664296093.188904] [28fd6574525b:81985:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:28:13.1895057Z [1664296093.188904] [28fd6574525b:81985:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:28:13.1895543Z [1664296093.189177] [28fd6574525b:81986:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:28:13.1896035Z [1664296093.189177] [28fd6574525b:81986:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:28:13.5929194Z STAGE:2022-09-27 16:28:13 81986:81986 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:28:13 81985:81985 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:13.5929651Z 
2022-09-27T16:28:13.7153440Z STAGE:2022-09-27 16:28:13 81985:81985 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:13.7180215Z STAGE:2022-09-27 16:28:13 81986:81986 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:13.7190736Z STAGE:2022-09-27 16:28:13 81985:81985 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:13.7191379Z STAGE:2022-09-27 16:28:13 81986:81986 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:14.9000404Z ok (7.535s)
2022-09-27T16:28:14.9000966Z 
2022-09-27T16:28:14.9001555Z ----------------------------------------------------------------------
2022-09-27T16:28:14.9002085Z Ran 1 test in 7.535s
2022-09-27T16:28:14.9002337Z 
2022-09-27T16:28:14.9002455Z OK
2022-09-27T16:28:14.9002662Z 
2022-09-27T16:28:14.9002862Z Generating XML reports...
2022-09-27T16:28:14.9036011Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162807.xml
2022-09-27T16:28:15.4402607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:15.4403140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:15.4405245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:15.4405746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:15.6093521Z ok (7.231s)
2022-09-27T16:28:15.6093732Z 
2022-09-27T16:28:15.6094126Z ----------------------------------------------------------------------
2022-09-27T16:28:15.6094450Z Ran 1 test in 7.231s
2022-09-27T16:28:15.6094618Z 
2022-09-27T16:28:15.6094716Z OK
2022-09-27T16:28:15.6094853Z 
2022-09-27T16:28:15.6094997Z Generating XML reports...
2022-09-27T16:28:15.6130036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162808.xml
2022-09-27T16:28:15.9655658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:28:15.9696328Z 
2022-09-27T16:28:15.9696521Z Running tests...
2022-09-27T16:28:15.9697187Z ----------------------------------------------------------------------
2022-09-27T16:28:17.1080568Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:17.1081740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:17.1082939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:17.1083886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:17.6080996Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:17.6104959Z 
2022-09-27T16:28:17.6105458Z Running tests...
2022-09-27T16:28:17.6106331Z ----------------------------------------------------------------------
2022-09-27T16:28:17.6128174Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.002s)
2022-09-27T16:28:17.6128909Z 
2022-09-27T16:28:17.6129451Z ----------------------------------------------------------------------
2022-09-27T16:28:17.6130069Z Ran 1 test in 0.002s
2022-09-27T16:28:17.6130400Z 
2022-09-27T16:28:17.6130609Z OK (skipped=1)
2022-09-27T16:28:17.6130913Z 
2022-09-27T16:28:17.6131129Z Generating XML reports...
2022-09-27T16:28:17.6166584Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162817.xml
2022-09-27T16:28:17.6359214Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:17.6559132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82330
2022-09-27T16:28:17.6564577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82331
2022-09-27T16:28:17.7575318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:17.7576243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:17.7578181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:17.7579097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:18.2536789Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:18.2559143Z 
2022-09-27T16:28:18.2559443Z Running tests...
2022-09-27T16:28:18.2560581Z ----------------------------------------------------------------------
2022-09-27T16:28:18.2578364Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:28:18.2579012Z 
2022-09-27T16:28:18.2579610Z ----------------------------------------------------------------------
2022-09-27T16:28:18.2580065Z Ran 1 test in 0.002s
2022-09-27T16:28:18.2580214Z 
2022-09-27T16:28:18.2580325Z OK (skipped=1)
2022-09-27T16:28:18.2580483Z 
2022-09-27T16:28:18.2580613Z Generating XML reports...
2022-09-27T16:28:18.2614709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162818.xml
2022-09-27T16:28:19.3989740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:19.3990626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:19.3991231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:19.3991714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:19.4212978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:19.4213475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:19.4216464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:19.4216984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:19.7067980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:19.7068526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:19.7070989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:19.7071488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:19.9387717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:19.9388273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:19.9389020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:19.9389738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:19.9396050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:19.9396843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:19.9493202Z STAGE:2022-09-27 16:28:19 82330:82330 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:19.9493937Z STAGE:2022-09-27 16:28:19 82331:82331 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:19.9508830Z STAGE:2022-09-27 16:28:19 82330:82330 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:28:19 82331:82331 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:19.9509243Z 
2022-09-27T16:28:19.9525492Z STAGE:2022-09-27 16:28:19 82330:82330 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:19.9526272Z STAGE:2022-09-27 16:28:19 82331:82331 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:19.9540150Z STAGE:2022-09-27 16:28:19 82330:82330 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:28:19 82331:82331 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:19.9540558Z 
2022-09-27T16:28:20.2203684Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:20.2226982Z 
2022-09-27T16:28:20.2227129Z Running tests...
2022-09-27T16:28:20.2227776Z ----------------------------------------------------------------------
2022-09-27T16:28:20.2254532Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.003s)
2022-09-27T16:28:20.2254920Z 
2022-09-27T16:28:20.2255244Z ----------------------------------------------------------------------
2022-09-27T16:28:20.2255605Z Ran 1 test in 0.003s
2022-09-27T16:28:20.2255776Z 
2022-09-27T16:28:20.2255889Z OK (skipped=1)
2022-09-27T16:28:20.2256032Z 
2022-09-27T16:28:20.2256159Z Generating XML reports...
2022-09-27T16:28:20.2298343Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162820.xml
2022-09-27T16:28:20.2414690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:20.2415301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:20.2418062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:20.2418556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:20.3631717Z ok (4.393s)
2022-09-27T16:28:20.3632115Z 
2022-09-27T16:28:20.3632785Z ----------------------------------------------------------------------
2022-09-27T16:28:20.3633444Z Ran 1 test in 4.393s
2022-09-27T16:28:20.3633764Z 
2022-09-27T16:28:20.3633934Z OK
2022-09-27T16:28:20.3634194Z 
2022-09-27T16:28:20.3634440Z Generating XML reports...
2022-09-27T16:28:20.3670318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162815.xml
2022-09-27T16:28:20.7403701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:20.7434692Z 
2022-09-27T16:28:20.7434970Z Running tests...
2022-09-27T16:28:20.7435394Z ----------------------------------------------------------------------
2022-09-27T16:28:22.2555222Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:22.2555867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:22.2557152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:22.2557648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:22.4838137Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:22.5036048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82542
2022-09-27T16:28:22.5042320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82543
2022-09-27T16:28:22.6567908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:22.6568474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:22.6570456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:22.6571253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:22.7622548Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:22.7646694Z 
2022-09-27T16:28:22.7647128Z Running tests...
2022-09-27T16:28:22.7647650Z ----------------------------------------------------------------------
2022-09-27T16:28:22.7674002Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.003s)
2022-09-27T16:28:22.7674414Z 
2022-09-27T16:28:22.7674704Z ----------------------------------------------------------------------
2022-09-27T16:28:22.7675024Z Ran 1 test in 0.003s
2022-09-27T16:28:22.7675188Z 
2022-09-27T16:28:22.7675298Z OK (skipped=1)
2022-09-27T16:28:22.7675454Z 
2022-09-27T16:28:22.7675581Z Generating XML reports...
2022-09-27T16:28:22.7717996Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162822.xml
2022-09-27T16:28:23.1503595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:28:23.1529829Z 
2022-09-27T16:28:23.1531352Z Running tests...
2022-09-27T16:28:23.1534992Z ----------------------------------------------------------------------
2022-09-27T16:28:24.1882885Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:24.1883820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:24.1885323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:24.1885813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:24.2191456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:24.2191962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:24.2195295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:24.2195794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:24.7002240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:24.7401758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:24.7402583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:24.7407362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:24.7409985Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:24.7413814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:24.7610345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82648
2022-09-27T16:28:24.7615713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82649
2022-09-27T16:28:24.7777414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:24.7777902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:24.7779993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:24.7780486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:24.8416099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:25.2113999Z skip: Skipped due to small world size. (4.467s)
2022-09-27T16:28:25.2114460Z 
2022-09-27T16:28:25.2115100Z ----------------------------------------------------------------------
2022-09-27T16:28:25.2115709Z Ran 1 test in 4.468s
2022-09-27T16:28:25.2116009Z 
2022-09-27T16:28:25.2116576Z OK (skipped=1)
2022-09-27T16:28:25.2116888Z 
2022-09-27T16:28:25.2117124Z Generating XML reports...
2022-09-27T16:28:25.2151029Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162820.xml
2022-09-27T16:28:25.2922927Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:25.2945512Z 
2022-09-27T16:28:25.2945800Z Running tests...
2022-09-27T16:28:25.2946239Z ----------------------------------------------------------------------
2022-09-27T16:28:25.2964930Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:28:25.2965230Z 
2022-09-27T16:28:25.2965509Z ----------------------------------------------------------------------
2022-09-27T16:28:25.2965845Z Ran 1 test in 0.002s
2022-09-27T16:28:25.2966014Z 
2022-09-27T16:28:25.2966131Z OK (skipped=1)
2022-09-27T16:28:25.2966290Z 
2022-09-27T16:28:25.2966419Z Generating XML reports...
2022-09-27T16:28:25.3000338Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162825.xml
2022-09-27T16:28:26.3527345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:26.3528764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:26.3533353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:26.3534340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:26.4641570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:26.4642074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:26.4645275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:26.4645789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:27.0010928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:27.0011523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:27.0012312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:27.0013017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:27.0119811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:27.0187626Z STAGE:2022-09-27 16:28:27 82649:82649 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:27.1023390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:27.1087778Z STAGE:2022-09-27 16:28:27 82648:82648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:27.1102458Z STAGE:2022-09-27 16:28:27 82648:82648 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:27.1103070Z STAGE:2022-09-27 16:28:27 82649:82649 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:27.1120142Z STAGE:2022-09-27 16:28:27 82648:82648 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:27.1120986Z STAGE:2022-09-27 16:28:27 82649:82649 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:27.1134757Z STAGE:2022-09-27 16:28:27 82648:82648 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:28:27 82649:82649 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:27.1135168Z 
2022-09-27T16:28:27.2670639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:27.2671174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:27.2673470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:27.2673975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:27.3994058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:27.3994549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:27.3996561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:27.3997046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:27.4688889Z ok (4.316s)
2022-09-27T16:28:27.4689093Z 
2022-09-27T16:28:27.4689493Z ----------------------------------------------------------------------
2022-09-27T16:28:27.4689828Z Ran 1 test in 4.316s
2022-09-27T16:28:27.4690000Z 
2022-09-27T16:28:27.4690083Z OK
2022-09-27T16:28:27.4690228Z 
2022-09-27T16:28:27.4690363Z Generating XML reports...
2022-09-27T16:28:27.4727884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162823.xml
2022-09-27T16:28:27.7771662Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:27.7793340Z 
2022-09-27T16:28:27.7793627Z Running tests...
2022-09-27T16:28:27.7794059Z ----------------------------------------------------------------------
2022-09-27T16:28:27.7812493Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:28:27.7812809Z 
2022-09-27T16:28:27.7813091Z ----------------------------------------------------------------------
2022-09-27T16:28:27.7813427Z Ran 1 test in 0.002s
2022-09-27T16:28:27.7813597Z 
2022-09-27T16:28:27.7813723Z OK (skipped=1)
2022-09-27T16:28:27.7813862Z 
2022-09-27T16:28:27.7813995Z Generating XML reports...
2022-09-27T16:28:27.7847862Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162827.xml
2022-09-27T16:28:27.8903394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:27.8926240Z 
2022-09-27T16:28:27.8926747Z Running tests...
2022-09-27T16:28:27.8927257Z ----------------------------------------------------------------------
2022-09-27T16:28:27.8944670Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:28:27.8945092Z 
2022-09-27T16:28:27.8945441Z ----------------------------------------------------------------------
2022-09-27T16:28:27.8945778Z Ran 1 test in 0.002s
2022-09-27T16:28:27.8945951Z 
2022-09-27T16:28:27.8946064Z OK (skipped=1)
2022-09-27T16:28:27.8946229Z 
2022-09-27T16:28:27.8946379Z Generating XML reports...
2022-09-27T16:28:27.8981955Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162827.xml
2022-09-27T16:28:29.6614607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:29.6615134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:29.6618638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:29.6619123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:29.7337477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:29.7337974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:29.7341092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:29.7341595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:29.8906200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:29.8906714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:29.8909518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:29.8909992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:30.1523109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:28:30.1546335Z 
2022-09-27T16:28:30.1546556Z Running tests...
2022-09-27T16:28:30.1546999Z ----------------------------------------------------------------------
2022-09-27T16:28:30.2120803Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:30.2143815Z 
2022-09-27T16:28:30.2144208Z Running tests...
2022-09-27T16:28:30.2144916Z ----------------------------------------------------------------------
2022-09-27T16:28:30.2163934Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T16:28:30.2164246Z 
2022-09-27T16:28:30.2164538Z ----------------------------------------------------------------------
2022-09-27T16:28:30.2164867Z Ran 1 test in 0.002s
2022-09-27T16:28:30.2165033Z 
2022-09-27T16:28:30.2165129Z OK (skipped=1)
2022-09-27T16:28:30.2165849Z 
2022-09-27T16:28:30.2166051Z Generating XML reports...
2022-09-27T16:28:30.2200345Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162830.xml
2022-09-27T16:28:30.3942642Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:30.3967099Z 
2022-09-27T16:28:30.3967485Z Running tests...
2022-09-27T16:28:30.3968170Z ----------------------------------------------------------------------
2022-09-27T16:28:30.3987156Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:28:30.3987808Z 
2022-09-27T16:28:30.3988086Z ----------------------------------------------------------------------
2022-09-27T16:28:30.3988432Z Ran 1 test in 0.002s
2022-09-27T16:28:30.3988598Z 
2022-09-27T16:28:30.3988710Z OK (skipped=1)
2022-09-27T16:28:30.3988866Z 
2022-09-27T16:28:30.3988993Z Generating XML reports...
2022-09-27T16:28:30.4026053Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162830.xml
2022-09-27T16:28:31.8149427Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:31.8342348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82959
2022-09-27T16:28:31.8348108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82960
2022-09-27T16:28:32.2529867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:32.2530437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:32.2531401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:32.2531886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:32.4550538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:32.4552413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:32.4555310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:32.4556928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:32.7983193Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:32.8006789Z 
2022-09-27T16:28:32.8007368Z Running tests...
2022-09-27T16:28:32.8007842Z ----------------------------------------------------------------------
2022-09-27T16:28:32.8026867Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T16:28:32.8027209Z 
2022-09-27T16:28:32.8027540Z ----------------------------------------------------------------------
2022-09-27T16:28:32.8027881Z Ran 1 test in 0.002s
2022-09-27T16:28:32.8028051Z 
2022-09-27T16:28:32.8028150Z OK (skipped=1)
2022-09-27T16:28:32.8028310Z 
2022-09-27T16:28:32.8028437Z Generating XML reports...
2022-09-27T16:28:32.8064011Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162832.xml
2022-09-27T16:28:33.0316554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:33.0338530Z 
2022-09-27T16:28:33.0339011Z Running tests...
2022-09-27T16:28:33.0339749Z ----------------------------------------------------------------------
2022-09-27T16:28:33.4764135Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:33.4764744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:33.4767445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:33.4767945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:33.5287562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:33.5288067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:33.5290313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:33.5290818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:34.1307871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:34.1308422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:34.1309198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:34.1309890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:34.1416612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:34.2322624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:34.6627692Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:34.6796602Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2022-09-27T16:28:34.6797250Z 
2022-09-27T16:28:34.6797513Z ----------------------------------------------------------------------
2022-09-27T16:28:34.6797848Z Ran 1 test in 1.646s
2022-09-27T16:28:34.6798013Z 
2022-09-27T16:28:34.6798121Z OK (skipped=1)
2022-09-27T16:28:34.6798277Z 
2022-09-27T16:28:34.6798400Z Generating XML reports...
2022-09-27T16:28:34.6830941Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162833.xml
2022-09-27T16:28:34.8080944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:34.8081673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:34.8082780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:34.8083253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:35.3283512Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:35.3307501Z 
2022-09-27T16:28:35.3307740Z Running tests...
2022-09-27T16:28:35.3308355Z ----------------------------------------------------------------------
2022-09-27T16:28:35.3325793Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:28:35.3327536Z 
2022-09-27T16:28:35.3328039Z ----------------------------------------------------------------------
2022-09-27T16:28:35.3328461Z Ran 1 test in 0.002s
2022-09-27T16:28:35.3328631Z 
2022-09-27T16:28:35.3328728Z OK (skipped=1)
2022-09-27T16:28:35.3328887Z 
2022-09-27T16:28:35.3329021Z Generating XML reports...
2022-09-27T16:28:35.3363488Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162835.xml
2022-09-27T16:28:35.7404613Z STAGE:2022-09-27 16:28:35 82960:82960 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:35.8479965Z STAGE:2022-09-27 16:28:35 82959:82959 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:35.8531389Z STAGE:2022-09-27 16:28:35 82960:82960 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:35.8534992Z STAGE:2022-09-27 16:28:35 82959:82959 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:35.8634002Z STAGE:2022-09-27 16:28:35 82960:82960 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:35.8651883Z STAGE:2022-09-27 16:28:35 82959:82959 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:35.8696211Z STAGE:2022-09-27 16:28:35 82960:82960 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:35.8697754Z STAGE:2022-09-27 16:28:35 82959:82959 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:36.4453377Z ok (6.290s)
2022-09-27T16:28:36.4453763Z 
2022-09-27T16:28:36.4454334Z ----------------------------------------------------------------------
2022-09-27T16:28:36.4454948Z Ran 1 test in 6.290s
2022-09-27T16:28:36.4455141Z 
2022-09-27T16:28:36.4455235Z OK
2022-09-27T16:28:36.4455375Z 
2022-09-27T16:28:36.4455495Z Generating XML reports...
2022-09-27T16:28:36.4491089Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162830.xml
2022-09-27T16:28:36.7971455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:36.7971979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:36.7974070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:36.7974575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:37.2888421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:37.2910525Z 
2022-09-27T16:28:37.2911114Z Running tests...
2022-09-27T16:28:37.2912273Z ----------------------------------------------------------------------
2022-09-27T16:28:37.3589222Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:37.3590422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:37.3592817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:37.3593821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:37.8842820Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:37.8865962Z 
2022-09-27T16:28:37.8866227Z Running tests...
2022-09-27T16:28:37.8866681Z ----------------------------------------------------------------------
2022-09-27T16:28:37.8885453Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T16:28:37.8886002Z 
2022-09-27T16:28:37.8886459Z ----------------------------------------------------------------------
2022-09-27T16:28:37.8886817Z Ran 1 test in 0.002s
2022-09-27T16:28:37.8886987Z 
2022-09-27T16:28:37.8887109Z OK (skipped=1)
2022-09-27T16:28:37.8887266Z 
2022-09-27T16:28:37.8887397Z Generating XML reports...
2022-09-27T16:28:37.8920590Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162837.xml
2022-09-27T16:28:38.4408911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:38.4410046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:38.4411121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:38.4411643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:38.9280139Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:28:38.9303115Z 
2022-09-27T16:28:38.9303378Z Running tests...
2022-09-27T16:28:38.9303833Z ----------------------------------------------------------------------
2022-09-27T16:28:38.9327847Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:38.9528251Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83208
2022-09-27T16:28:38.9535283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83209
2022-09-27T16:28:39.8995939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:39.8996719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:39.8998546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:39.8999267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:40.4121340Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:40.4144882Z 
2022-09-27T16:28:40.4145429Z Running tests...
2022-09-27T16:28:40.4146006Z ----------------------------------------------------------------------
2022-09-27T16:28:40.4164422Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:28:40.4164749Z 
2022-09-27T16:28:40.4165049Z ----------------------------------------------------------------------
2022-09-27T16:28:40.4165606Z Ran 1 test in 0.002s
2022-09-27T16:28:40.4165792Z 
2022-09-27T16:28:40.4165901Z OK (skipped=1)
2022-09-27T16:28:40.4166057Z 
2022-09-27T16:28:40.4166181Z Generating XML reports...
2022-09-27T16:28:40.4201421Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162840.xml
2022-09-27T16:28:40.5468874Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:40.5668500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83275
2022-09-27T16:28:40.5673733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83276
2022-09-27T16:28:40.5997147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:40.5997698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:40.5998724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:40.5999224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:40.6429541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:40.6431613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:40.6432496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:40.6432990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:41.1034564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:41.1464173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:41.1465009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:41.1541449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:41.1548143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:41.2542273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:42.2713096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:42.2713836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:42.2714658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:42.2715153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:42.3233594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:42.3234326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:42.3235521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:42.3236007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:42.4728392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:42.4728912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:42.4731329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:42.4731858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:42.8785754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:42.8786366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:42.8787169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:42.8787879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:42.8896376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:42.8989020Z STAGE:2022-09-27 16:28:42 83209:83209 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:42.9121121Z STAGE:2022-09-27 16:28:42 83208:83208 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:42.9799253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:42.9987540Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:43.0012388Z 
2022-09-27T16:28:43.0012653Z Running tests...
2022-09-27T16:28:43.0013089Z ----------------------------------------------------------------------
2022-09-27T16:28:43.0033123Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:28:43.0033732Z 
2022-09-27T16:28:43.0034200Z ----------------------------------------------------------------------
2022-09-27T16:28:43.0034626Z Ran 1 test in 0.002s
2022-09-27T16:28:43.0034793Z 
2022-09-27T16:28:43.0034904Z OK (skipped=1)
2022-09-27T16:28:43.0035061Z 
2022-09-27T16:28:43.0035191Z Generating XML reports...
2022-09-27T16:28:43.0072308Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162842.xml
2022-09-27T16:28:44.4376391Z STAGE:2022-09-27 16:28:44 83208:83208 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:44.4377032Z STAGE:2022-09-27 16:28:44 83209:83209 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:44.5504900Z STAGE:2022-09-27 16:28:44 83276:83276 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:44.6556951Z STAGE:2022-09-27 16:28:44 83275:83275 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:44.6606917Z STAGE:2022-09-27 16:28:44 83275:83275 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:44.6607546Z STAGE:2022-09-27 16:28:44 83276:83276 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:44.6668467Z STAGE:2022-09-27 16:28:44 83276:83276 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:28:44 83275:83275 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:44.6668874Z 
2022-09-27T16:28:44.6691248Z STAGE:2022-09-27 16:28:44 83275:83275 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:44.6691865Z STAGE:2022-09-27 16:28:44 83276:83276 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:45.0753675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:45.0754569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:45.0756223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:45.0757151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:45.1775961Z ok (6.247s)
2022-09-27T16:28:45.1776182Z 
2022-09-27T16:28:45.1776530Z ----------------------------------------------------------------------
2022-09-27T16:28:45.1776874Z Ran 1 test in 6.247s
2022-09-27T16:28:45.1777056Z 
2022-09-27T16:28:45.1777149Z OK
2022-09-27T16:28:45.1777286Z 
2022-09-27T16:28:45.1777424Z Generating XML reports...
2022-09-27T16:28:45.1814880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162838.xml
2022-09-27T16:28:45.3681330Z ok (8.077s)
2022-09-27T16:28:45.3681546Z 
2022-09-27T16:28:45.3681907Z ----------------------------------------------------------------------
2022-09-27T16:28:45.3682242Z Ran 1 test in 8.077s
2022-09-27T16:28:45.3682390Z 
2022-09-27T16:28:45.3682487Z OK
2022-09-27T16:28:45.3682621Z 
2022-09-27T16:28:45.3682755Z Generating XML reports...
2022-09-27T16:28:45.3718708Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162837.xml
2022-09-27T16:28:45.5799055Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:45.5822879Z 
2022-09-27T16:28:45.5823346Z Running tests...
2022-09-27T16:28:45.5824707Z ----------------------------------------------------------------------
2022-09-27T16:28:45.5842134Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:28:45.5842501Z 
2022-09-27T16:28:45.5842778Z ----------------------------------------------------------------------
2022-09-27T16:28:45.5843347Z Ran 1 test in 0.002s
2022-09-27T16:28:45.5843520Z 
2022-09-27T16:28:45.5843631Z OK (skipped=1)
2022-09-27T16:28:45.5843786Z 
2022-09-27T16:28:45.5843910Z Generating XML reports...
2022-09-27T16:28:45.5878644Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162845.xml
2022-09-27T16:28:47.3507752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:47.3508706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:47.3510300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:47.3511274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:47.5534566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:47.5535549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:47.5537499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:47.5538446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:47.5707999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:47.5708967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:47.5711519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:47.5712505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:47.8291648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:28:47.8312995Z 
2022-09-27T16:28:47.8313347Z Running tests...
2022-09-27T16:28:47.8314198Z ----------------------------------------------------------------------
2022-09-27T16:28:48.0562308Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:48.0563724Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:48.0583692Z 
2022-09-27T16:28:48.0584204Z Running tests...
2022-09-27T16:28:48.0585024Z ----------------------------------------------------------------------
2022-09-27T16:28:48.0586750Z 
2022-09-27T16:28:48.0587097Z Running tests...
2022-09-27T16:28:48.0587938Z ----------------------------------------------------------------------
2022-09-27T16:28:48.0608210Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:28:48.0608914Z 
2022-09-27T16:28:48.0609456Z ----------------------------------------------------------------------
2022-09-27T16:28:48.0610108Z Ran 1 test in 0.002s
2022-09-27T16:28:48.0610413Z 
2022-09-27T16:28:48.0610618Z OK (skipped=1)
2022-09-27T16:28:48.0610922Z 
2022-09-27T16:28:48.0611158Z Generating XML reports...
2022-09-27T16:28:48.0622660Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.004s)
2022-09-27T16:28:48.0623020Z 
2022-09-27T16:28:48.0623352Z ----------------------------------------------------------------------
2022-09-27T16:28:48.0623690Z Ran 1 test in 0.004s
2022-09-27T16:28:48.0623858Z 
2022-09-27T16:28:48.0623969Z OK (skipped=1)
2022-09-27T16:28:48.0624382Z 
2022-09-27T16:28:48.0624507Z Generating XML reports...
2022-09-27T16:28:48.0648074Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162848.xml
2022-09-27T16:28:48.0658780Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162848.xml
2022-09-27T16:28:49.4363683Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:49.4554359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83611
2022-09-27T16:28:49.4560633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83612
2022-09-27T16:28:50.0269169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:50.0269884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:50.0271207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:50.0271696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:50.1565772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:50.1567423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:50.1571675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:50.1572783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:50.5665608Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:50.5688236Z 
2022-09-27T16:28:50.5688505Z Running tests...
2022-09-27T16:28:50.5688930Z ----------------------------------------------------------------------
2022-09-27T16:28:50.6985930Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:50.7009522Z 
2022-09-27T16:28:50.7009788Z Running tests...
2022-09-27T16:28:50.7010227Z ----------------------------------------------------------------------
2022-09-27T16:28:50.7029503Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:28:50.7030000Z 
2022-09-27T16:28:50.7030397Z ----------------------------------------------------------------------
2022-09-27T16:28:50.7030735Z Ran 1 test in 0.002s
2022-09-27T16:28:50.7030883Z 
2022-09-27T16:28:50.7030993Z OK (skipped=1)
2022-09-27T16:28:50.7031148Z 
2022-09-27T16:28:50.7031277Z Generating XML reports...
2022-09-27T16:28:50.7066004Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162850.xml
2022-09-27T16:28:51.0529249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:51.0529746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:51.0530934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:51.0531424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:51.0965480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:51.0965953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:51.0967908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:51.0968400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:51.6169370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:51.6169972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:51.6170763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:51.6171489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:51.6277226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:51.6343763Z STAGE:2022-09-27 16:28:51 83612:83612 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:51.7190487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:51.7300715Z STAGE:2022-09-27 16:28:51 83611:83611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:51.7333543Z STAGE:2022-09-27 16:28:51 83611:83611 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:28:51 83612:83612 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:51.7335253Z 
2022-09-27T16:28:51.7367323Z STAGE:2022-09-27 16:28:51 83612:83612 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:51.7379638Z STAGE:2022-09-27 16:28:51 83611:83611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:28:51.7410527Z STAGE:2022-09-27 16:28:51 83611:83611 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:51.7418382Z STAGE:2022-09-27 16:28:51 83612:83612 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:28:52.1212111Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:52.1377096Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.569s)
2022-09-27T16:28:52.1377675Z 
2022-09-27T16:28:52.1377953Z ----------------------------------------------------------------------
2022-09-27T16:28:52.1378297Z Ran 1 test in 1.569s
2022-09-27T16:28:52.1378462Z 
2022-09-27T16:28:52.1378572Z OK (skipped=1)
2022-09-27T16:28:52.1378714Z 
2022-09-27T16:28:52.1378840Z Generating XML reports...
2022-09-27T16:28:52.1412166Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162850.xml
2022-09-27T16:28:52.1627569Z ok (4.331s)
2022-09-27T16:28:52.1627766Z 
2022-09-27T16:28:52.1628109Z ----------------------------------------------------------------------
2022-09-27T16:28:52.1628431Z Ran 1 test in 4.331s
2022-09-27T16:28:52.1628607Z 
2022-09-27T16:28:52.1628704Z OK
2022-09-27T16:28:52.1628843Z 
2022-09-27T16:28:52.1628977Z Generating XML reports...
2022-09-27T16:28:52.1667201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162847.xml
2022-09-27T16:28:52.6607665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:52.6608375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:52.6611106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:52.6611618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:53.1454319Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:53.1476426Z 
2022-09-27T16:28:53.1476699Z Running tests...
2022-09-27T16:28:53.1477407Z ----------------------------------------------------------------------
2022-09-27T16:28:53.1496343Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:28:53.1496745Z 
2022-09-27T16:28:53.1497268Z ----------------------------------------------------------------------
2022-09-27T16:28:53.1497650Z Ran 1 test in 0.002s
2022-09-27T16:28:53.1497829Z 
2022-09-27T16:28:53.1497939Z OK (skipped=1)
2022-09-27T16:28:53.1498095Z 
2022-09-27T16:28:53.1498221Z Generating XML reports...
2022-09-27T16:28:53.1531782Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162853.xml
2022-09-27T16:28:54.3208970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:54.3209492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:54.3210653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:54.3211128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:54.4085666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:54.4086170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:54.4088682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:54.4089178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:54.8265576Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:54.8289152Z 
2022-09-27T16:28:54.8289352Z Running tests...
2022-09-27T16:28:54.8289794Z ----------------------------------------------------------------------
2022-09-27T16:28:54.8308300Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:28:54.8308634Z 
2022-09-27T16:28:54.8308941Z ----------------------------------------------------------------------
2022-09-27T16:28:54.8309277Z Ran 1 test in 0.002s
2022-09-27T16:28:54.8309443Z 
2022-09-27T16:28:54.8309553Z OK (skipped=1)
2022-09-27T16:28:54.8309715Z 
2022-09-27T16:28:54.8309834Z Generating XML reports...
2022-09-27T16:28:54.8345278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162854.xml
2022-09-27T16:28:54.8913689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:28:54.8936400Z 
2022-09-27T16:28:54.8936887Z Running tests...
2022-09-27T16:28:54.8937401Z ----------------------------------------------------------------------
2022-09-27T16:28:55.1452783Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:55.1453814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:55.1455098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:55.1455598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:55.6399783Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:55.6422285Z 
2022-09-27T16:28:55.6422514Z Running tests...
2022-09-27T16:28:55.6422965Z ----------------------------------------------------------------------
2022-09-27T16:28:55.6440495Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:28:55.6440977Z 
2022-09-27T16:28:55.6441621Z ----------------------------------------------------------------------
2022-09-27T16:28:55.6441983Z Ran 1 test in 0.002s
2022-09-27T16:28:55.6442142Z 
2022-09-27T16:28:55.6442254Z OK (skipped=1)
2022-09-27T16:28:55.6442413Z 
2022-09-27T16:28:55.6442540Z Generating XML reports...
2022-09-27T16:28:55.6477174Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162855.xml
2022-09-27T16:28:56.5269245Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:56.5467350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83859
2022-09-27T16:28:56.5474480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83860
2022-09-27T16:28:56.7915308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:56.7915895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:56.7917214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:56.7917708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:57.3423215Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:28:57.3445411Z 
2022-09-27T16:28:57.3445705Z Running tests...
2022-09-27T16:28:57.3446213Z ----------------------------------------------------------------------
2022-09-27T16:28:57.7055723Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:57.7056346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:57.7058181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:57.7058656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:58.1821798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:58.1822735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:58.1823861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:58.1824812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:58.1848609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:28:58.1849504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:28:58.1852265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:28:58.1867016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:28:58.2266957Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:28:58.2290747Z 
2022-09-27T16:28:58.2290912Z Running tests...
2022-09-27T16:28:58.2291371Z ----------------------------------------------------------------------
2022-09-27T16:28:58.2310323Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:28:58.2310705Z 
2022-09-27T16:28:58.2310990Z ----------------------------------------------------------------------
2022-09-27T16:28:58.2311325Z Ran 1 test in 0.002s
2022-09-27T16:28:58.2311493Z 
2022-09-27T16:28:58.2311609Z OK (skipped=1)
2022-09-27T16:28:58.2311771Z 
2022-09-27T16:28:58.2311880Z Generating XML reports...
2022-09-27T16:28:58.2348032Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162858.xml
2022-09-27T16:28:58.7326376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:28:58.7326958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:28:58.7327992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:58.7328741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:28:58.7434870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:28:58.8340172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:28:58.8707508Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:28:58.8907115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83971
2022-09-27T16:28:58.8912083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83972
2022-09-27T16:29:00.2809756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:00.2810289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:00.2812580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:00.2813065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:00.5905136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:00.5905681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:00.5906890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:00.5907529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:00.6238968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:00.6239646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:00.6241203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:00.6241689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:00.7967089Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:00.7990864Z 
2022-09-27T16:29:00.7991216Z Running tests...
2022-09-27T16:29:00.7991717Z ----------------------------------------------------------------------
2022-09-27T16:29:00.8010856Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:29:00.8011344Z 
2022-09-27T16:29:00.8011778Z ----------------------------------------------------------------------
2022-09-27T16:29:00.8012403Z Ran 1 test in 0.002s
2022-09-27T16:29:00.8012605Z 
2022-09-27T16:29:00.8012717Z OK (skipped=1)
2022-09-27T16:29:00.8012872Z 
2022-09-27T16:29:00.8013001Z Generating XML reports...
2022-09-27T16:29:00.8048217Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162900.xml
2022-09-27T16:29:00.8575629Z ok (5.964s)
2022-09-27T16:29:00.8575954Z 
2022-09-27T16:29:00.8576491Z ----------------------------------------------------------------------
2022-09-27T16:29:00.8576842Z Ran 1 test in 5.964s
2022-09-27T16:29:00.8577011Z 
2022-09-27T16:29:00.8577107Z OK
2022-09-27T16:29:00.8577250Z 
2022-09-27T16:29:00.8577380Z Generating XML reports...
2022-09-27T16:29:00.8612827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162854.xml
2022-09-27T16:29:01.1021694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:01.1510991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:01.1511802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:01.1529074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:01.1536298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:01.2524418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:01.5982289Z skip: Skipped due to small world size. (4.253s)
2022-09-27T16:29:01.5982547Z 
2022-09-27T16:29:01.5983157Z ----------------------------------------------------------------------
2022-09-27T16:29:01.5983534Z Ran 1 test in 4.254s
2022-09-27T16:29:01.5983709Z 
2022-09-27T16:29:01.5983812Z OK (skipped=1)
2022-09-27T16:29:01.5983963Z 
2022-09-27T16:29:01.5984092Z Generating XML reports...
2022-09-27T16:29:01.6019291Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162857.xml
2022-09-27T16:29:02.8810465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:02.8811477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:02.8817268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:02.8818493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:03.0061193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:03.0061708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:03.0065084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:03.0065580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:03.3875025Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:03.3899470Z 
2022-09-27T16:29:03.3900010Z Running tests...
2022-09-27T16:29:03.3900623Z ----------------------------------------------------------------------
2022-09-27T16:29:03.3920393Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:29:03.3921462Z 
2022-09-27T16:29:03.3922044Z ----------------------------------------------------------------------
2022-09-27T16:29:03.3922472Z Ran 1 test in 0.002s
2022-09-27T16:29:03.3922640Z 
2022-09-27T16:29:03.3922749Z OK (skipped=1)
2022-09-27T16:29:03.3922905Z 
2022-09-27T16:29:03.3923030Z Generating XML reports...
2022-09-27T16:29:03.3956163Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162903.xml
2022-09-27T16:29:03.4908288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:29:03.4931995Z 
2022-09-27T16:29:03.4932399Z Running tests...
2022-09-27T16:29:03.4932966Z ----------------------------------------------------------------------
2022-09-27T16:29:03.7629203Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:03.7629806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:03.7631271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:03.7631758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:04.2599205Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:04.2621774Z 
2022-09-27T16:29:04.2621924Z Running tests...
2022-09-27T16:29:04.2622764Z ----------------------------------------------------------------------
2022-09-27T16:29:04.2641869Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:29:04.2642416Z 
2022-09-27T16:29:04.2642766Z ----------------------------------------------------------------------
2022-09-27T16:29:04.2643118Z Ran 1 test in 0.002s
2022-09-27T16:29:04.2643289Z 
2022-09-27T16:29:04.2643401Z OK (skipped=1)
2022-09-27T16:29:04.2643560Z 
2022-09-27T16:29:04.2643687Z Generating XML reports...
2022-09-27T16:29:04.2677548Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162904.xml
2022-09-27T16:29:05.0396325Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:05.0913468Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84209
2022-09-27T16:29:05.0917846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84210
2022-09-27T16:29:05.3510299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:05.3512048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:05.3514795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:05.3516408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:05.9633442Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:05.9657787Z 
2022-09-27T16:29:05.9658112Z Running tests...
2022-09-27T16:29:05.9658615Z ----------------------------------------------------------------------
2022-09-27T16:29:05.9676484Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:29:05.9677242Z 
2022-09-27T16:29:05.9677560Z ----------------------------------------------------------------------
2022-09-27T16:29:05.9677910Z Ran 1 test in 0.002s
2022-09-27T16:29:05.9678099Z 
2022-09-27T16:29:05.9678213Z OK (skipped=1)
2022-09-27T16:29:05.9678373Z 
2022-09-27T16:29:05.9678501Z Generating XML reports...
2022-09-27T16:29:05.9713300Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162905.xml
2022-09-27T16:29:06.3024725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:06.3025252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:06.3027782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:06.3028287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:06.7728311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:06.7728810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:06.7730002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:06.7730495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:06.7855111Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:06.7878099Z 
2022-09-27T16:29:06.7878479Z Running tests...
2022-09-27T16:29:06.7878901Z ----------------------------------------------------------------------
2022-09-27T16:29:06.7896549Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:29:06.7897512Z 
2022-09-27T16:29:06.7897873Z ----------------------------------------------------------------------
2022-09-27T16:29:06.7898520Z Ran 1 test in 0.002s
2022-09-27T16:29:06.7898853Z 
2022-09-27T16:29:06.7899086Z OK (skipped=1)
2022-09-27T16:29:06.7899388Z 
2022-09-27T16:29:06.7899864Z Generating XML reports...
2022-09-27T16:29:06.7932872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162906.xml
2022-09-27T16:29:06.8305030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:06.8305652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:06.8306982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:06.8307817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:07.4290527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:07.4291568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:07.4292389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:07.4293101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:07.4398177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:07.4463737Z STAGE:2022-09-27 16:29:07 84210:84210 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:07.5302783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:07.5369268Z STAGE:2022-09-27 16:29:07 84209:84209 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:07.5384115Z STAGE:2022-09-27 16:29:07 84209:84209 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:07.5385251Z STAGE:2022-09-27 16:29:07 84210:84210 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:07.5401231Z STAGE:2022-09-27 16:29:07 84209:84209 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:07.5402520Z STAGE:2022-09-27 16:29:07 84210:84210 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:07.5416360Z STAGE:2022-09-27 16:29:07 84209:84209 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:07.5417351Z STAGE:2022-09-27 16:29:07 84210:84210 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:07.8929514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:07.8930042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:07.8931569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:07.8932368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:07.9987189Z ok (4.505s)
2022-09-27T16:29:07.9987581Z 
2022-09-27T16:29:07.9988338Z ----------------------------------------------------------------------
2022-09-27T16:29:07.9988921Z Ran 1 test in 4.505s
2022-09-27T16:29:07.9989094Z 
2022-09-27T16:29:07.9989200Z OK
2022-09-27T16:29:07.9989338Z 
2022-09-27T16:29:07.9989479Z Generating XML reports...
2022-09-27T16:29:08.0024053Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162903.xml
2022-09-27T16:29:08.3854256Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:08.3875483Z 
2022-09-27T16:29:08.3875716Z Running tests...
2022-09-27T16:29:08.3876644Z ----------------------------------------------------------------------
2022-09-27T16:29:08.3894601Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:29:08.3895346Z 
2022-09-27T16:29:08.3896229Z ----------------------------------------------------------------------
2022-09-27T16:29:08.3896615Z Ran 1 test in 0.002s
2022-09-27T16:29:08.3896787Z 
2022-09-27T16:29:08.3896880Z OK (skipped=1)
2022-09-27T16:29:08.3897044Z 
2022-09-27T16:29:08.3897177Z Generating XML reports...
2022-09-27T16:29:08.3930354Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162908.xml
2022-09-27T16:29:08.7270406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:08.7270918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:08.7272236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:08.7272735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:09.2792771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:09.2815948Z 
2022-09-27T16:29:09.2816362Z Running tests...
2022-09-27T16:29:09.2816811Z ----------------------------------------------------------------------
2022-09-27T16:29:10.0899049Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:10.0900128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:10.0900828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:10.0901295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:10.3995717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:10.3996222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:10.3998503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:10.3999007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:10.5783046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:29:10.5805213Z 
2022-09-27T16:29:10.5805691Z Running tests...
2022-09-27T16:29:10.5806170Z ----------------------------------------------------------------------
2022-09-27T16:29:10.9073277Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:10.9096889Z 
2022-09-27T16:29:10.9097243Z Running tests...
2022-09-27T16:29:10.9097673Z ----------------------------------------------------------------------
2022-09-27T16:29:10.9117546Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:29:10.9117934Z 
2022-09-27T16:29:10.9118441Z ----------------------------------------------------------------------
2022-09-27T16:29:10.9118808Z Ran 1 test in 0.002s
2022-09-27T16:29:10.9118981Z 
2022-09-27T16:29:10.9119098Z OK (skipped=1)
2022-09-27T16:29:10.9119237Z 
2022-09-27T16:29:10.9119367Z Generating XML reports...
2022-09-27T16:29:10.9155072Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162910.xml
2022-09-27T16:29:10.9253343Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:10.9462041Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84455
2022-09-27T16:29:10.9467923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84456
2022-09-27T16:29:12.1973738Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:12.2168333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84552
2022-09-27T16:29:12.2174952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84553
2022-09-27T16:29:12.6201589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:12.6202155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:12.6203268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:12.6203760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:12.6269988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:12.6270469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:12.6274035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:12.6274529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:12.9422020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:12.9429381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:12.9442371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:12.9443346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:13.1931747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:13.1935425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:13.1936453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:13.2032126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:13.2038721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:13.2949863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:13.4529520Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:13.4554224Z 
2022-09-27T16:29:13.4554540Z Running tests...
2022-09-27T16:29:13.4554993Z ----------------------------------------------------------------------
2022-09-27T16:29:13.4574798Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:29:13.4575399Z 
2022-09-27T16:29:13.4575693Z ----------------------------------------------------------------------
2022-09-27T16:29:13.4576015Z Ran 1 test in 0.002s
2022-09-27T16:29:13.4576244Z 
2022-09-27T16:29:13.4576437Z OK (skipped=1)
2022-09-27T16:29:13.4576783Z 
2022-09-27T16:29:13.4576946Z Generating XML reports...
2022-09-27T16:29:13.4613239Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162913.xml
2022-09-27T16:29:13.8532900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:13.8533444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:13.8534451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:13.8535065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:13.8574006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:13.8574638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:13.8578469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:13.8579167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:14.3982564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:14.3983391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:14.3984409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:14.3985608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:14.3990687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:14.3991200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:14.4087045Z STAGE:2022-09-27 16:29:14 84552:84552 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:14.4087672Z STAGE:2022-09-27 16:29:14 84553:84553 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:14.4101954Z STAGE:2022-09-27 16:29:14 84552:84552 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:29:14 84553:84553 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:14.4102364Z 
2022-09-27T16:29:14.4119238Z STAGE:2022-09-27 16:29:14 84552:84552 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:29:14 84553:84553 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:14.4119631Z 
2022-09-27T16:29:14.4132617Z STAGE:2022-09-27 16:29:14 84553:84553 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:14.4133228Z STAGE:2022-09-27 16:29:14 84552:84552 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:14.8240995Z ok (4.243s)
2022-09-27T16:29:14.8241297Z 
2022-09-27T16:29:14.8242017Z ----------------------------------------------------------------------
2022-09-27T16:29:14.8242692Z Ran 1 test in 4.243s
2022-09-27T16:29:14.8242865Z 
2022-09-27T16:29:14.8242981Z OK
2022-09-27T16:29:14.8243119Z 
2022-09-27T16:29:14.8243260Z Generating XML reports...
2022-09-27T16:29:14.8277388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162910.xml
2022-09-27T16:29:15.4494323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:15.4494885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:15.4496453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:15.4497276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:15.9654975Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:15.9681481Z 
2022-09-27T16:29:15.9682286Z Running tests...
2022-09-27T16:29:15.9683197Z ----------------------------------------------------------------------
2022-09-27T16:29:15.9707519Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:29:15.9708377Z 
2022-09-27T16:29:15.9709088Z ----------------------------------------------------------------------
2022-09-27T16:29:15.9709841Z Ran 1 test in 0.003s
2022-09-27T16:29:15.9710211Z 
2022-09-27T16:29:15.9710462Z OK (skipped=1)
2022-09-27T16:29:15.9710853Z 
2022-09-27T16:29:15.9711108Z Generating XML reports...
2022-09-27T16:29:15.9753422Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162915.xml
2022-09-27T16:29:16.3595089Z ok (7.078s)
2022-09-27T16:29:16.3595352Z 
2022-09-27T16:29:16.3596058Z ----------------------------------------------------------------------
2022-09-27T16:29:16.3596426Z Ran 1 test in 7.078s
2022-09-27T16:29:16.3596598Z 
2022-09-27T16:29:16.3596695Z OK
2022-09-27T16:29:16.3596812Z 
2022-09-27T16:29:16.3596961Z Generating XML reports...
2022-09-27T16:29:16.3632838Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162909.xml
2022-09-27T16:29:16.9354399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:16.9355571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:16.9356952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:16.9358060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:17.4637652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:29:17.4682275Z 
2022-09-27T16:29:17.4682843Z Running tests...
2022-09-27T16:29:17.4683699Z ----------------------------------------------------------------------
2022-09-27T16:29:17.8682708Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:17.8684242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:17.8686635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:17.8687560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:18.3604096Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:18.3628467Z 
2022-09-27T16:29:18.3628962Z Running tests...
2022-09-27T16:29:18.3629474Z ----------------------------------------------------------------------
2022-09-27T16:29:18.3649001Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:29:18.3650063Z 
2022-09-27T16:29:18.3650386Z ----------------------------------------------------------------------
2022-09-27T16:29:18.3650736Z Ran 1 test in 0.002s
2022-09-27T16:29:18.3650905Z 
2022-09-27T16:29:18.3651020Z OK (skipped=1)
2022-09-27T16:29:18.3651183Z 
2022-09-27T16:29:18.3651311Z Generating XML reports...
2022-09-27T16:29:18.3685978Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162918.xml
2022-09-27T16:29:18.4198662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:18.4199170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:18.4200905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:18.4201379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:18.9203062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:18.9224904Z 
2022-09-27T16:29:18.9225338Z Running tests...
2022-09-27T16:29:18.9225820Z ----------------------------------------------------------------------
2022-09-27T16:29:19.0779593Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:19.0979627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84787
2022-09-27T16:29:19.0985804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84788
2022-09-27T16:29:20.4448487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:20.4449668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:20.4450341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:20.4450836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:20.5826633Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:20.6020200Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84884
2022-09-27T16:29:20.6026504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84885
2022-09-27T16:29:20.7942418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:20.7943248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:20.7944548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:20.7945138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:20.8656683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:20.8657398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:20.8658827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:20.8659333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:20.9534030Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:20.9557812Z 
2022-09-27T16:29:20.9558020Z Running tests...
2022-09-27T16:29:20.9558713Z ----------------------------------------------------------------------
2022-09-27T16:29:20.9577217Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:29:20.9577962Z 
2022-09-27T16:29:20.9578467Z ----------------------------------------------------------------------
2022-09-27T16:29:20.9579220Z Ran 1 test in 0.002s
2022-09-27T16:29:20.9579558Z 
2022-09-27T16:29:20.9579748Z OK (skipped=1)
2022-09-27T16:29:20.9579922Z 
2022-09-27T16:29:20.9580053Z Generating XML reports...
2022-09-27T16:29:20.9615898Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162920.xml
2022-09-27T16:29:21.4654216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:21.4655493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:21.4656032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:21.4656702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:21.4664413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:21.4769644Z STAGE:2022-09-27 16:29:21 84788:84788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:21.5665925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:21.5734077Z STAGE:2022-09-27 16:29:21 84787:84787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:21.5749626Z STAGE:2022-09-27 16:29:21 84787:84787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:21.5752932Z STAGE:2022-09-27 16:29:21 84788:84788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:21.5767523Z STAGE:2022-09-27 16:29:21 84787:84787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:21.5778885Z STAGE:2022-09-27 16:29:21 84788:84788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:21.5794070Z STAGE:2022-09-27 16:29:21 84787:84787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:21.5796223Z STAGE:2022-09-27 16:29:21 84788:84788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:22.0081621Z ok (4.538s)
2022-09-27T16:29:22.0090139Z 
2022-09-27T16:29:22.0094679Z ----------------------------------------------------------------------
2022-09-27T16:29:22.0095954Z Ran 1 test in 4.538s
2022-09-27T16:29:22.0096347Z 
2022-09-27T16:29:22.0096561Z OK
2022-09-27T16:29:22.0097185Z 
2022-09-27T16:29:22.0097508Z Generating XML reports...
2022-09-27T16:29:22.0135991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162917.xml
2022-09-27T16:29:22.2908752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:22.2909274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:22.2910372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:22.2910848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:22.3537398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:22.3537914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:22.3539322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:22.3539796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:22.8015285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:22.8504796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:22.8505630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:22.8521841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:22.8529083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:22.9518059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:23.0625958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:23.0626461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:23.0628267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:23.0628769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:23.6603395Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:23.6626915Z 
2022-09-27T16:29:23.6627176Z Running tests...
2022-09-27T16:29:23.6627595Z ----------------------------------------------------------------------
2022-09-27T16:29:23.6647903Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:29:23.6648264Z 
2022-09-27T16:29:23.6648553Z ----------------------------------------------------------------------
2022-09-27T16:29:23.6648892Z Ran 1 test in 0.002s
2022-09-27T16:29:23.6649061Z 
2022-09-27T16:29:23.6649151Z OK (skipped=1)
2022-09-27T16:29:23.6649311Z 
2022-09-27T16:29:23.6649439Z Generating XML reports...
2022-09-27T16:29:23.6684843Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162923.xml
2022-09-27T16:29:24.1999889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:24.2001367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:24.2002504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:24.2003421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:24.7027363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:29:24.7053020Z 
2022-09-27T16:29:24.7053818Z Running tests...
2022-09-27T16:29:24.7054885Z ----------------------------------------------------------------------
2022-09-27T16:29:25.6764775Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:25.6765415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:25.6766585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:25.6767082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:25.9156418Z ok (6.993s)
2022-09-27T16:29:25.9156679Z 
2022-09-27T16:29:25.9157071Z ----------------------------------------------------------------------
2022-09-27T16:29:25.9157420Z Ran 1 test in 6.993s
2022-09-27T16:29:25.9157592Z 
2022-09-27T16:29:25.9157693Z OK
2022-09-27T16:29:25.9157818Z 
2022-09-27T16:29:25.9157963Z Generating XML reports...
2022-09-27T16:29:25.9194001Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162918.xml
2022-09-27T16:29:26.1884865Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:26.1909439Z 
2022-09-27T16:29:26.1909779Z Running tests...
2022-09-27T16:29:26.1910250Z ----------------------------------------------------------------------
2022-09-27T16:29:26.3853912Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:26.4053494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85085
2022-09-27T16:29:26.4059370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85086
2022-09-27T16:29:27.8663807Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:27.8861825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85181
2022-09-27T16:29:27.8867769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85182
2022-09-27T16:29:28.0307252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:28.0307992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:28.0308607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:28.0309326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:28.0373850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:28.0374338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:28.0378567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:28.0379070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:28.0864841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:28.0865398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:28.0866883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:28.0867845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:28.5531457Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:28.5567959Z 
2022-09-27T16:29:28.5568694Z Running tests...
2022-09-27T16:29:28.5569769Z ----------------------------------------------------------------------
2022-09-27T16:29:28.5597201Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.003s)
2022-09-27T16:29:28.5597881Z 
2022-09-27T16:29:28.5598939Z ----------------------------------------------------------------------
2022-09-27T16:29:28.5599613Z Ran 1 test in 0.003s
2022-09-27T16:29:28.5599936Z 
2022-09-27T16:29:28.5600776Z OK (skipped=1)
2022-09-27T16:29:28.5601167Z 
2022-09-27T16:29:28.5601433Z Generating XML reports...
2022-09-27T16:29:28.5655929Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162928.xml
2022-09-27T16:29:28.6693046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:28.6694100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:28.6695842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:28.6697171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:28.6702575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:28.6703087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:29.5244054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:29.5245116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:29.5246613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:29.5247582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:29.5277040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:29.5278021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:29.5285170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:29.5286180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:30.0560003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:30.1133429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:30.1134274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:30.1170014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:30.1177028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:30.2146466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:30.2665732Z STAGE:2022-09-27 16:29:30 85086:85086 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:30.2777147Z STAGE:2022-09-27 16:29:30 85085:85085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:30.2825349Z STAGE:2022-09-27 16:29:30 85085:85085 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:30.2826265Z STAGE:2022-09-27 16:29:30 85086:85086 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:30.2877931Z STAGE:2022-09-27 16:29:30 85085:85085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:30.2878537Z STAGE:2022-09-27 16:29:30 85086:85086 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:30.5546560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:30.5547111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:30.5549806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:30.5550301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:31.0874249Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:31.0899697Z 
2022-09-27T16:29:31.0900141Z Running tests...
2022-09-27T16:29:31.0900669Z ----------------------------------------------------------------------
2022-09-27T16:29:31.7411766Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:29:31 85086:85086 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:29:31 85085:85085 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:31.7413266Z 
2022-09-27T16:29:31.7431929Z STAGE:2022-09-27 16:29:31 85085:85085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:31.7433069Z STAGE:2022-09-27 16:29:31 85086:85086 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:31.7456503Z STAGE:2022-09-27 16:29:31 85085:85085 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:31.7457719Z STAGE:2022-09-27 16:29:31 85086:85086 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:32.7197316Z ok (8.014s)
2022-09-27T16:29:32.7197557Z 
2022-09-27T16:29:32.7197983Z ----------------------------------------------------------------------
2022-09-27T16:29:32.7198346Z Ran 1 test in 8.014s
2022-09-27T16:29:32.7198520Z 
2022-09-27T16:29:32.7198624Z OK
2022-09-27T16:29:32.7198756Z 
2022-09-27T16:29:32.7198891Z Generating XML reports...
2022-09-27T16:29:32.7234033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162924.xml
2022-09-27T16:29:32.7249960Z [1664296172.724702] [28fd6574525b:85181:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:29:32.7250441Z [1664296172.724702] [28fd6574525b:85181:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:29:32.7324476Z [1664296172.732079] [28fd6574525b:85182:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:29:32.7324987Z [1664296172.732079] [28fd6574525b:85182:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:29:32.8203959Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:32.8413161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85310
2022-09-27T16:29:32.8417810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85311
2022-09-27T16:29:33.1031289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:29:33.1036016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:29:33.1037121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:29:33.1132746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:29:33.9014478Z ok (7.710s)
2022-09-27T16:29:33.9014720Z 
2022-09-27T16:29:33.9015377Z ----------------------------------------------------------------------
2022-09-27T16:29:33.9015731Z Ran 1 test in 7.710s
2022-09-27T16:29:33.9015901Z 
2022-09-27T16:29:33.9015998Z OK
2022-09-27T16:29:33.9016141Z 
2022-09-27T16:29:33.9016286Z Generating XML reports...
2022-09-27T16:29:33.9072519Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162926.xml
2022-09-27T16:29:34.5980495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:34.5981424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:34.5982524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:34.5983426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:34.6034647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:34.6035540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:34.6038232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:34.6039525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:34.9717074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:34.9718054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:34.9719206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:34.9720463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:35.1743523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:35.1750785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:35.1753414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:35.1755388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:35.1762584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:35.1767953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:29:35.2752747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:35.2756109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:29:35.2756936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:29:35.2783077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:29:35.5180345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:29:35.5202886Z 
2022-09-27T16:29:35.5203245Z Running tests...
2022-09-27T16:29:35.5203728Z ----------------------------------------------------------------------
2022-09-27T16:29:36.1313599Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:36.1314222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:36.1315206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:36.1315701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:36.6526418Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:36.6548895Z 
2022-09-27T16:29:36.6549319Z Running tests...
2022-09-27T16:29:36.6549837Z ----------------------------------------------------------------------
2022-09-27T16:29:36.6568399Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:29:36.6568743Z 
2022-09-27T16:29:36.6569020Z ----------------------------------------------------------------------
2022-09-27T16:29:36.6569340Z Ran 1 test in 0.002s
2022-09-27T16:29:36.6569510Z 
2022-09-27T16:29:36.6569621Z OK (skipped=1)
2022-09-27T16:29:36.6569780Z 
2022-09-27T16:29:36.6569907Z Generating XML reports...
2022-09-27T16:29:36.6604597Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162936.xml
2022-09-27T16:29:37.1461023Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:37.1662877Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85476
2022-09-27T16:29:37.1668332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85477
2022-09-27T16:29:38.4539466Z ok (7.364s)
2022-09-27T16:29:38.4539688Z 
2022-09-27T16:29:38.4540089Z ----------------------------------------------------------------------
2022-09-27T16:29:38.4540416Z Ran 1 test in 7.364s
2022-09-27T16:29:38.4540584Z 
2022-09-27T16:29:38.4540685Z OK
2022-09-27T16:29:38.4540822Z 
2022-09-27T16:29:38.4540959Z Generating XML reports...
2022-09-27T16:29:38.4576025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162931.xml
2022-09-27T16:29:38.7548534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:38.7549038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:38.7551513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:38.7552023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:38.9220622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:38.9221143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:38.9221961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:38.9222451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:38.9306025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:38.9306492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:38.9310075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:38.9310565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:39.2557152Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:39.2580504Z 
2022-09-27T16:29:39.2580833Z Running tests...
2022-09-27T16:29:39.2581298Z ----------------------------------------------------------------------
2022-09-27T16:29:39.2599926Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:29:39.2600291Z 
2022-09-27T16:29:39.2600794Z ----------------------------------------------------------------------
2022-09-27T16:29:39.2601119Z Ran 1 test in 0.002s
2022-09-27T16:29:39.2601288Z 
2022-09-27T16:29:39.2601399Z OK (skipped=1)
2022-09-27T16:29:39.2601558Z 
2022-09-27T16:29:39.2601689Z Generating XML reports...
2022-09-27T16:29:39.2635594Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162939.xml
2022-09-27T16:29:39.4632186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:39.4632771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:39.4633553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:39.4634260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:39.4640801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:39.4641298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:40.6383161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:40.6383717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:40.6386190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:40.6387041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:41.1622463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:41.1647916Z 
2022-09-27T16:29:41.1648557Z Running tests...
2022-09-27T16:29:41.1649222Z ----------------------------------------------------------------------
2022-09-27T16:29:41.1668083Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:29:41.1668450Z 
2022-09-27T16:29:41.1668741Z ----------------------------------------------------------------------
2022-09-27T16:29:41.1669055Z Ran 1 test in 0.002s
2022-09-27T16:29:41.1669224Z 
2022-09-27T16:29:41.1669358Z OK (skipped=1)
2022-09-27T16:29:41.1669519Z 
2022-09-27T16:29:41.1669649Z Generating XML reports...
2022-09-27T16:29:41.1706353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162941.xml
2022-09-27T16:29:41.2014298Z STAGE:2022-09-27 16:29:41 85476:85476 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:41.2413340Z STAGE:2022-09-27 16:29:41 85477:85477 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:41.2461156Z STAGE:2022-09-27 16:29:41 85477:85477 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:41.2461772Z STAGE:2022-09-27 16:29:41 85476:85476 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:41.2514528Z STAGE:2022-09-27 16:29:41 85477:85477 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:41.2515535Z STAGE:2022-09-27 16:29:41 85476:85476 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:41.2700642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:41.2701166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:41.2702467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:41.2702941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:41.7874735Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:41.7896793Z 
2022-09-27T16:29:41.7897213Z Running tests...
2022-09-27T16:29:41.7897702Z ----------------------------------------------------------------------
2022-09-27T16:29:41.7914606Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-09-27T16:29:41.7914903Z 
2022-09-27T16:29:41.7915487Z ----------------------------------------------------------------------
2022-09-27T16:29:41.7915849Z Ran 1 test in 0.002s
2022-09-27T16:29:41.7916019Z 
2022-09-27T16:29:41.7916132Z OK (skipped=1)
2022-09-27T16:29:41.7916292Z 
2022-09-27T16:29:41.7916412Z Generating XML reports...
2022-09-27T16:29:41.7950432Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162941.xml
2022-09-27T16:29:42.6143718Z STAGE:2022-09-27 16:29:42 85476:85476 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:29:42 85477:85477 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:42.6144160Z 
2022-09-27T16:29:42.6164978Z STAGE:2022-09-27 16:29:42 85477:85477 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:42.6165696Z STAGE:2022-09-27 16:29:42 85476:85476 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:42.6186463Z STAGE:2022-09-27 16:29:42 85476:85476 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:43.1469836Z STAGE:2022-09-27 16:29:42 85477:85477 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:43.1470632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:43.1471680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:43.1473002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:43.1473712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:43.6765022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:43.6788766Z 
2022-09-27T16:29:43.6789309Z Running tests...
2022-09-27T16:29:43.6789823Z ----------------------------------------------------------------------
2022-09-27T16:29:43.6809523Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... ok (8.160s)
2022-09-27T16:29:43.6810087Z 
2022-09-27T16:29:43.6810643Z ----------------------------------------------------------------------
2022-09-27T16:29:43.6811288Z Ran 1 test in 8.161s
2022-09-27T16:29:43.6811582Z 
2022-09-27T16:29:43.6811713Z OK
2022-09-27T16:29:43.6811948Z 
2022-09-27T16:29:43.6812170Z Generating XML reports...
2022-09-27T16:29:43.6850241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162935.xml
2022-09-27T16:29:43.7403358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:43.7404302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:43.7405517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:43.7406365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:44.2653517Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:44.2675405Z 
2022-09-27T16:29:44.2675550Z Running tests...
2022-09-27T16:29:44.2677085Z ----------------------------------------------------------------------
2022-09-27T16:29:44.2696169Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... skip: flaky on PyTorch CI with timeout (0.002s)
2022-09-27T16:29:44.2696824Z 
2022-09-27T16:29:44.2697395Z ----------------------------------------------------------------------
2022-09-27T16:29:44.2698032Z Ran 1 test in 0.002s
2022-09-27T16:29:44.2698368Z 
2022-09-27T16:29:44.2698587Z OK (skipped=1)
2022-09-27T16:29:44.2698906Z 
2022-09-27T16:29:44.2699167Z Generating XML reports...
2022-09-27T16:29:44.2732395Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162944.xml
2022-09-27T16:29:45.3670667Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:45.3872249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85795
2022-09-27T16:29:45.3879705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85796
2022-09-27T16:29:45.8282728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:45.8283240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:45.8286024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:45.8286523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:46.2864512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:46.2865454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:46.2866619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:46.2867560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:46.3788830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:29:46.3813589Z 
2022-09-27T16:29:46.3814058Z Running tests...
2022-09-27T16:29:46.3814571Z ----------------------------------------------------------------------
2022-09-27T16:29:46.7804572Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:46.7827320Z 
2022-09-27T16:29:46.7827760Z Running tests...
2022-09-27T16:29:46.7828199Z ----------------------------------------------------------------------
2022-09-27T16:29:46.7847250Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-09-27T16:29:46.7847566Z 
2022-09-27T16:29:46.7848176Z ----------------------------------------------------------------------
2022-09-27T16:29:46.7848564Z Ran 1 test in 0.002s
2022-09-27T16:29:46.7848720Z 
2022-09-27T16:29:46.7848840Z OK (skipped=1)
2022-09-27T16:29:46.7849003Z 
2022-09-27T16:29:46.7849151Z Generating XML reports...
2022-09-27T16:29:46.7884154Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162946.xml
2022-09-27T16:29:47.0413916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:47.0414449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:47.0418619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:47.0419552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:47.0489392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:47.0489861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:47.0493183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:47.0493686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:47.5620830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:47.5768838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:47.5769655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:47.5823450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:47.5830667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:47.6783424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:48.0023167Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:48.0220204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85903
2022-09-27T16:29:48.0227844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85904
2022-09-27T16:29:48.0954779Z skip: Skipped due to small world size. (4.416s)
2022-09-27T16:29:48.0955029Z 
2022-09-27T16:29:48.0955422Z ----------------------------------------------------------------------
2022-09-27T16:29:48.0955779Z Ran 1 test in 4.417s
2022-09-27T16:29:48.0955948Z 
2022-09-27T16:29:48.0956058Z OK (skipped=1)
2022-09-27T16:29:48.0956196Z 
2022-09-27T16:29:48.0956329Z Generating XML reports...
2022-09-27T16:29:48.1009749Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162943.xml
2022-09-27T16:29:48.8667254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:48.8667772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:48.8670027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:48.8670525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:49.4585990Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:49.4609208Z 
2022-09-27T16:29:49.4609546Z Running tests...
2022-09-27T16:29:49.4610001Z ----------------------------------------------------------------------
2022-09-27T16:29:49.7012063Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:49.7012706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:49.7013570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:49.7014069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:49.7609376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:49.7610147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:49.7611803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:49.7612294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:50.2700901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:50.2701435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:50.2702468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:50.2702964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:50.2919395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:50.2919941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:50.2920935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:50.2921643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:50.3028332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:50.3936894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:50.7785637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:29:50.7809587Z 
2022-09-27T16:29:50.7809835Z Running tests...
2022-09-27T16:29:50.7810298Z ----------------------------------------------------------------------
2022-09-27T16:29:51.1618054Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:51.1830620Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86016
2022-09-27T16:29:51.1835802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86017
2022-09-27T16:29:51.9617279Z STAGE:2022-09-27 16:29:51 85904:85904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:52.0227829Z STAGE:2022-09-27 16:29:52 85903:85903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:52.0279610Z STAGE:2022-09-27 16:29:52 85903:85903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:52.0280572Z STAGE:2022-09-27 16:29:52 85904:85904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:52.0334152Z STAGE:2022-09-27 16:29:52 85903:85903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:52.0334989Z STAGE:2022-09-27 16:29:52 85904:85904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:52.4724226Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:29:52.4927626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86087
2022-09-27T16:29:52.4932361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86088
2022-09-27T16:29:52.8349517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:52.8350322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:52.8351203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:52.8351697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:52.8598800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:52.8599676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:52.8600946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:52.8601439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:53.4177600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:53.4361413Z STAGE:2022-09-27 16:29:53 85904:85904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:53.4384164Z STAGE:2022-09-27 16:29:53 85904:85904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:53.4503364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:53.4504059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:53.4521871Z STAGE:2022-09-27 16:29:53 85903:85903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:53.4545507Z STAGE:2022-09-27 16:29:53 85903:85903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:29:53.4567273Z STAGE:2022-09-27 16:29:53 85903:85903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:53.4567851Z STAGE:2022-09-27 16:29:53 85904:85904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:29:53.4582855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:53.4590959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:53.5518750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:53.9912727Z skip: Skipped due to small world size. (4.530s)
2022-09-27T16:29:53.9912978Z 
2022-09-27T16:29:53.9913375Z ----------------------------------------------------------------------
2022-09-27T16:29:53.9913698Z Ran 1 test in 4.530s
2022-09-27T16:29:53.9913863Z 
2022-09-27T16:29:53.9913973Z OK (skipped=1)
2022-09-27T16:29:53.9914131Z 
2022-09-27T16:29:53.9914261Z Generating XML reports...
2022-09-27T16:29:53.9950023Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162949.xml
2022-09-27T16:29:54.1568426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:54.1569133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:54.1570997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:54.1571515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:54.1703742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:54.1704244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:54.1707012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:54.1707789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:54.4361536Z ok (8.054s)
2022-09-27T16:29:54.4361903Z 
2022-09-27T16:29:54.4362339Z ----------------------------------------------------------------------
2022-09-27T16:29:54.4362669Z Ran 1 test in 8.055s
2022-09-27T16:29:54.4362837Z 
2022-09-27T16:29:54.4363012Z OK
2022-09-27T16:29:54.4363284Z 
2022-09-27T16:29:54.4363487Z Generating XML reports...
2022-09-27T16:29:54.4398835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162946.xml
2022-09-27T16:29:54.6801729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:29:54.6820037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:29:54.6820859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:54.6903778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:29:54.6910454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:29:54.7836522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:29:56.1838148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:56.1838673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:56.1840080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:56.1840920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:56.5469557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:56.5470109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:56.5472804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:56.5473304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:56.7035280Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:56.7059467Z 
2022-09-27T16:29:56.7060024Z Running tests...
2022-09-27T16:29:56.7060544Z ----------------------------------------------------------------------
2022-09-27T16:29:56.7079404Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-09-27T16:29:56.7080722Z 
2022-09-27T16:29:56.7081398Z ----------------------------------------------------------------------
2022-09-27T16:29:56.7081785Z Ran 1 test in 0.002s
2022-09-27T16:29:56.7081962Z 
2022-09-27T16:29:56.7082074Z OK (skipped=1)
2022-09-27T16:29:56.7082216Z 
2022-09-27T16:29:56.7082344Z Generating XML reports...
2022-09-27T16:29:56.7116289Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162956.xml
2022-09-27T16:29:57.0567711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:29:57.0590608Z 
2022-09-27T16:29:57.0590786Z Running tests...
2022-09-27T16:29:57.0591246Z ----------------------------------------------------------------------
2022-09-27T16:29:57.0611188Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:29:57.0611826Z 
2022-09-27T16:29:57.0612648Z ----------------------------------------------------------------------
2022-09-27T16:29:57.0613036Z Ran 1 test in 0.002s
2022-09-27T16:29:57.0613211Z 
2022-09-27T16:29:57.0613329Z OK (skipped=1)
2022-09-27T16:29:57.0613499Z 
2022-09-27T16:29:57.0613611Z Generating XML reports...
2022-09-27T16:29:57.0648702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162957.xml
2022-09-27T16:29:57.1765002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:29:57.1766138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:29:57.1766942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:29:57.1866818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:29:57.9060675Z ok (7.125s)
2022-09-27T16:29:57.9060882Z 
2022-09-27T16:29:57.9061288Z ----------------------------------------------------------------------
2022-09-27T16:29:57.9061633Z Ran 1 test in 7.125s
2022-09-27T16:29:57.9061799Z 
2022-09-27T16:29:57.9061877Z OK
2022-09-27T16:29:57.9062018Z 
2022-09-27T16:29:57.9062159Z Generating XML reports...
2022-09-27T16:29:57.9096878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162950.xml
2022-09-27T16:29:58.7971951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:58.7972935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:58.7974572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:58.7975517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:59.0746575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:59.0747584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:59.0752506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:59.0753470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:29:59.3603916Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:29:59.3627569Z 
2022-09-27T16:29:59.3627868Z Running tests...
2022-09-27T16:29:59.3628646Z ----------------------------------------------------------------------
2022-09-27T16:29:59.5683029Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:29:59.5705539Z 
2022-09-27T16:29:59.5705843Z Running tests...
2022-09-27T16:29:59.5706503Z ----------------------------------------------------------------------
2022-09-27T16:29:59.5725346Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:29:59.5725672Z 
2022-09-27T16:29:59.5725963Z ----------------------------------------------------------------------
2022-09-27T16:29:59.5726298Z Ran 1 test in 0.002s
2022-09-27T16:29:59.5726464Z 
2022-09-27T16:29:59.5726584Z OK (skipped=1)
2022-09-27T16:29:59.5726742Z 
2022-09-27T16:29:59.5726850Z Generating XML reports...
2022-09-27T16:29:59.5760519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162959.xml
2022-09-27T16:29:59.9664017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:29:59.9664869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:29:59.9666827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:29:59.9667332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:00.4627399Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:00.4652723Z 
2022-09-27T16:30:00.4653076Z Running tests...
2022-09-27T16:30:00.4653891Z ----------------------------------------------------------------------
2022-09-27T16:30:00.9685608Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:00.9890798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86384
2022-09-27T16:30:00.9896611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86385
2022-09-27T16:30:01.5215267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:01.5216330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:01.5217525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:01.5218490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:02.0877526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:02.0900080Z 
2022-09-27T16:30:02.0900396Z Running tests...
2022-09-27T16:30:02.0900828Z ----------------------------------------------------------------------
2022-09-27T16:30:02.0918433Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T16:30:02.0918744Z 
2022-09-27T16:30:02.0919902Z ----------------------------------------------------------------------
2022-09-27T16:30:02.0921140Z Ran 1 test in 0.002s
2022-09-27T16:30:02.0921360Z 
2022-09-27T16:30:02.0921469Z OK (skipped=1)
2022-09-27T16:30:02.0921638Z 
2022-09-27T16:30:02.0921767Z Generating XML reports...
2022-09-27T16:30:02.0956122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163002.xml
2022-09-27T16:30:02.0974524Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:02.1183078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86450
2022-09-27T16:30:02.1186842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86451
2022-09-27T16:30:02.6888532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:02.6889432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:02.6891080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:02.6891895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:02.6903837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:02.6904628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:02.6908390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:02.6909154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:03.3237857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:03.3281845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:03.3283273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:03.3338592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:03.3345137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:03.4296528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:03.7999637Z skip: Skipped due to small world size. (4.437s)
2022-09-27T16:30:03.7999897Z 
2022-09-27T16:30:03.8000558Z ----------------------------------------------------------------------
2022-09-27T16:30:03.8000918Z Ran 1 test in 4.437s
2022-09-27T16:30:03.8001086Z 
2022-09-27T16:30:03.8001197Z OK (skipped=1)
2022-09-27T16:30:03.8001361Z 
2022-09-27T16:30:03.8001492Z Generating XML reports...
2022-09-27T16:30:03.8036624Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162959.xml
2022-09-27T16:30:03.8331978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:03.8332503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:03.8334882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:03.8335362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:03.8492954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:03.8493447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:03.8495759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:03.8496251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:04.1183963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:04.1184472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:04.1185919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:04.1186404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:04.3563329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:04.3683837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:04.3684930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:04.3765640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:04.3772271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:04.4698589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:04.6212584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:04.6234908Z 
2022-09-27T16:30:04.6235251Z Running tests...
2022-09-27T16:30:04.6235684Z ----------------------------------------------------------------------
2022-09-27T16:30:04.6254756Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T16:30:04.6255322Z 
2022-09-27T16:30:04.6255770Z ----------------------------------------------------------------------
2022-09-27T16:30:04.6256137Z Ran 1 test in 0.002s
2022-09-27T16:30:04.6256314Z 
2022-09-27T16:30:04.6256454Z OK (skipped=1)
2022-09-27T16:30:04.6256615Z 
2022-09-27T16:30:04.6256725Z Generating XML reports...
2022-09-27T16:30:04.6291067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163004.xml
2022-09-27T16:30:04.8254685Z skip: Need at least 3 CUDA devices (4.360s)
2022-09-27T16:30:04.8254929Z 
2022-09-27T16:30:04.8255317Z ----------------------------------------------------------------------
2022-09-27T16:30:04.8255644Z Ran 1 test in 4.360s
2022-09-27T16:30:04.8255809Z 
2022-09-27T16:30:04.8255919Z OK (skipped=1)
2022-09-27T16:30:04.8256075Z 
2022-09-27T16:30:04.8256208Z Generating XML reports...
2022-09-27T16:30:04.8291384Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163000.xml
2022-09-27T16:30:06.1149412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:06.1150491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:06.1152306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:06.1153284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:06.6029926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:06.6030449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:06.6033013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:06.6033506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:06.6162398Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:06.6187161Z 
2022-09-27T16:30:06.6187425Z Running tests...
2022-09-27T16:30:06.6187881Z ----------------------------------------------------------------------
2022-09-27T16:30:06.6208042Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T16:30:06.6208637Z 
2022-09-27T16:30:06.6208950Z ----------------------------------------------------------------------
2022-09-27T16:30:06.6209475Z Ran 1 test in 0.002s
2022-09-27T16:30:06.6209665Z 
2022-09-27T16:30:06.6209777Z OK (skipped=1)
2022-09-27T16:30:06.6209936Z 
2022-09-27T16:30:06.6210061Z Generating XML reports...
2022-09-27T16:30:06.6244656Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163006.xml
2022-09-27T16:30:07.0024301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:07.0024980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:07.0026720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:07.0027488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:07.0968154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:07.0990508Z 
2022-09-27T16:30:07.0991013Z Running tests...
2022-09-27T16:30:07.0991514Z ----------------------------------------------------------------------
2022-09-27T16:30:07.1009883Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:30:07.1010205Z 
2022-09-27T16:30:07.1010495Z ----------------------------------------------------------------------
2022-09-27T16:30:07.1010835Z Ran 1 test in 0.002s
2022-09-27T16:30:07.1011006Z 
2022-09-27T16:30:07.1011101Z OK (skipped=1)
2022-09-27T16:30:07.1011259Z 
2022-09-27T16:30:07.1011393Z Generating XML reports...
2022-09-27T16:30:07.1045776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163007.xml
2022-09-27T16:30:07.4963556Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:07.4985641Z 
2022-09-27T16:30:07.4986041Z Running tests...
2022-09-27T16:30:07.4986672Z ----------------------------------------------------------------------
2022-09-27T16:30:07.5005003Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-09-27T16:30:07.5005469Z 
2022-09-27T16:30:07.5005881Z ----------------------------------------------------------------------
2022-09-27T16:30:07.5006201Z Ran 1 test in 0.002s
2022-09-27T16:30:07.5006374Z 
2022-09-27T16:30:07.5006486Z OK (skipped=1)
2022-09-27T16:30:07.5006719Z 
2022-09-27T16:30:07.5006957Z Generating XML reports...
2022-09-27T16:30:07.5039877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163007.xml
2022-09-27T16:30:08.5928115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:08.5937217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:08.5942387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:08.5947355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:09.0622517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:09.0623004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:09.0625589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:09.0626086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:09.1096109Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:09.1119350Z 
2022-09-27T16:30:09.1119495Z Running tests...
2022-09-27T16:30:09.1120611Z ----------------------------------------------------------------------
2022-09-27T16:30:09.1141938Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T16:30:09.1142277Z 
2022-09-27T16:30:09.1142761Z ----------------------------------------------------------------------
2022-09-27T16:30:09.1143192Z Ran 1 test in 0.002s
2022-09-27T16:30:09.1143344Z 
2022-09-27T16:30:09.1143456Z OK (skipped=1)
2022-09-27T16:30:09.1143613Z 
2022-09-27T16:30:09.1143743Z Generating XML reports...
2022-09-27T16:30:09.1177763Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163009.xml
2022-09-27T16:30:09.5276932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:09.5278128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:09.5279302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:09.5280563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:09.5625410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:09.5648022Z 
2022-09-27T16:30:09.5648422Z Running tests...
2022-09-27T16:30:09.5648952Z ----------------------------------------------------------------------
2022-09-27T16:30:09.5667942Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-09-27T16:30:09.5668289Z 
2022-09-27T16:30:09.5668616Z ----------------------------------------------------------------------
2022-09-27T16:30:09.5668941Z Ran 1 test in 0.002s
2022-09-27T16:30:09.5669108Z 
2022-09-27T16:30:09.5669230Z OK (skipped=1)
2022-09-27T16:30:09.5669411Z 
2022-09-27T16:30:09.5669542Z Generating XML reports...
2022-09-27T16:30:09.5703983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163009.xml
2022-09-27T16:30:10.0306824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:10.0329925Z 
2022-09-27T16:30:10.0330193Z Running tests...
2022-09-27T16:30:10.0330619Z ----------------------------------------------------------------------
2022-09-27T16:30:10.0349488Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-09-27T16:30:10.0349821Z 
2022-09-27T16:30:10.0350157Z ----------------------------------------------------------------------
2022-09-27T16:30:10.0350504Z Ran 1 test in 0.002s
2022-09-27T16:30:10.0350656Z 
2022-09-27T16:30:10.0350772Z OK (skipped=1)
2022-09-27T16:30:10.0350930Z 
2022-09-27T16:30:10.0351060Z Generating XML reports...
2022-09-27T16:30:10.0385942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163010.xml
2022-09-27T16:30:11.1820130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:11.1820670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:11.1822659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:11.1823157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:11.6230780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:11.6231275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:11.6233851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:11.6234359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:11.6738215Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:11.6761273Z 
2022-09-27T16:30:11.6761785Z Running tests...
2022-09-27T16:30:11.6762276Z ----------------------------------------------------------------------
2022-09-27T16:30:11.6781109Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T16:30:11.6781441Z 
2022-09-27T16:30:11.6781727Z ----------------------------------------------------------------------
2022-09-27T16:30:11.6782045Z Ran 1 test in 0.002s
2022-09-27T16:30:11.6782210Z 
2022-09-27T16:30:11.6782323Z OK (skipped=1)
2022-09-27T16:30:11.6782484Z 
2022-09-27T16:30:11.6784431Z Generating XML reports...
2022-09-27T16:30:11.6818014Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163011.xml
2022-09-27T16:30:12.0312035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:12.0312526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:12.0314734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:12.0315231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:12.1317015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:12.1340868Z 
2022-09-27T16:30:12.1341293Z Running tests...
2022-09-27T16:30:12.1341832Z ----------------------------------------------------------------------
2022-09-27T16:30:12.1360353Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-09-27T16:30:12.1361300Z 
2022-09-27T16:30:12.1361850Z ----------------------------------------------------------------------
2022-09-27T16:30:12.1362181Z Ran 1 test in 0.002s
2022-09-27T16:30:12.1362349Z 
2022-09-27T16:30:12.1362458Z OK (skipped=1)
2022-09-27T16:30:12.1362947Z 
2022-09-27T16:30:12.1363075Z Generating XML reports...
2022-09-27T16:30:12.1397232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163012.xml
2022-09-27T16:30:12.5183347Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:12.5205486Z 
2022-09-27T16:30:12.5206007Z Running tests...
2022-09-27T16:30:12.5206901Z ----------------------------------------------------------------------
2022-09-27T16:30:13.6491167Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:13.6492018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:13.6494420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:13.6494900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:14.1218110Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:14.1286516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:14.1287000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:14.1290229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:14.1290715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:14.1410933Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86924
2022-09-27T16:30:14.1417493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86925
2022-09-27T16:30:14.1542273Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:14.1567530Z 
2022-09-27T16:30:14.1567742Z Running tests...
2022-09-27T16:30:14.1568192Z ----------------------------------------------------------------------
2022-09-27T16:30:14.1590866Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-09-27T16:30:14.1592687Z 
2022-09-27T16:30:14.1593421Z ----------------------------------------------------------------------
2022-09-27T16:30:14.1593796Z Ran 1 test in 0.002s
2022-09-27T16:30:14.1593967Z 
2022-09-27T16:30:14.1594091Z OK (skipped=1)
2022-09-27T16:30:14.1594236Z 
2022-09-27T16:30:14.1594364Z Generating XML reports...
2022-09-27T16:30:14.1628891Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163014.xml
2022-09-27T16:30:14.6322358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:14.6344370Z 
2022-09-27T16:30:14.6344703Z Running tests...
2022-09-27T16:30:14.6345194Z ----------------------------------------------------------------------
2022-09-27T16:30:14.6363879Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:14.6364214Z 
2022-09-27T16:30:14.6364500Z ----------------------------------------------------------------------
2022-09-27T16:30:14.6364831Z Ran 1 test in 0.002s
2022-09-27T16:30:14.6364998Z 
2022-09-27T16:30:14.6365097Z OK (skipped=1)
2022-09-27T16:30:14.6365257Z 
2022-09-27T16:30:14.6365386Z Generating XML reports...
2022-09-27T16:30:14.6400432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163014.xml
2022-09-27T16:30:15.8813793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:15.8814509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:15.8815403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:15.8816165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:15.8987972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:15.8988431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:15.8991442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:15.8991932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:16.2391861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:16.2392468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:16.2394204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:16.2394692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:16.3897218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:16.3999577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:16.4000665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:16.4099354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:16.4105774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:16.5014344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:16.6997009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:16.6997911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:16.6998917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:16.6999405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:16.7404714Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:16.7426853Z 
2022-09-27T16:30:16.7427311Z Running tests...
2022-09-27T16:30:16.7427825Z ----------------------------------------------------------------------
2022-09-27T16:30:16.7450968Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-09-27T16:30:16.7451386Z 
2022-09-27T16:30:16.7452195Z ----------------------------------------------------------------------
2022-09-27T16:30:16.7452563Z Ran 1 test in 0.002s
2022-09-27T16:30:16.7452730Z 
2022-09-27T16:30:16.7452843Z OK (skipped=1)
2022-09-27T16:30:16.7453010Z 
2022-09-27T16:30:16.7453120Z Generating XML reports...
2022-09-27T16:30:16.7487056Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163016.xml
2022-09-27T16:30:17.2323335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:17.2347111Z 
2022-09-27T16:30:17.2347535Z Running tests...
2022-09-27T16:30:17.2348021Z ----------------------------------------------------------------------
2022-09-27T16:30:17.2367095Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:30:17.2367449Z 
2022-09-27T16:30:17.2367774Z ----------------------------------------------------------------------
2022-09-27T16:30:17.2368147Z Ran 1 test in 0.002s
2022-09-27T16:30:17.2368315Z 
2022-09-27T16:30:17.2368426Z OK (skipped=1)
2022-09-27T16:30:17.2368569Z 
2022-09-27T16:30:17.2368698Z Generating XML reports...
2022-09-27T16:30:17.2405752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163017.xml
2022-09-27T16:30:18.7604191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:18.7605114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:18.7606272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:18.7607218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:19.2398293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:19.2399399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:19.2401095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:19.2402231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:19.2601263Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:19.2625691Z 
2022-09-27T16:30:19.2626074Z Running tests...
2022-09-27T16:30:19.2626993Z ----------------------------------------------------------------------
2022-09-27T16:30:19.2650184Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:30:19.2650897Z 
2022-09-27T16:30:19.2651692Z ----------------------------------------------------------------------
2022-09-27T16:30:19.2652075Z Ran 1 test in 0.002s
2022-09-27T16:30:19.2652243Z 
2022-09-27T16:30:19.2652350Z OK (skipped=1)
2022-09-27T16:30:19.2652526Z 
2022-09-27T16:30:19.2652656Z Generating XML reports...
2022-09-27T16:30:19.2686253Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163019.xml
2022-09-27T16:30:19.7268018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:19.7289887Z 
2022-09-27T16:30:19.7290197Z Running tests...
2022-09-27T16:30:19.7290645Z ----------------------------------------------------------------------
2022-09-27T16:30:19.7308694Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:30:19.7309049Z 
2022-09-27T16:30:19.7309340Z ----------------------------------------------------------------------
2022-09-27T16:30:19.7309676Z Ran 1 test in 0.002s
2022-09-27T16:30:19.7309841Z 
2022-09-27T16:30:19.7309952Z OK (skipped=1)
2022-09-27T16:30:19.7310091Z 
2022-09-27T16:30:19.7310217Z Generating XML reports...
2022-09-27T16:30:19.7344418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163019.xml
2022-09-27T16:30:20.2561100Z ok (7.735s)
2022-09-27T16:30:20.2562096Z 
2022-09-27T16:30:20.2563650Z ----------------------------------------------------------------------
2022-09-27T16:30:20.2564468Z Ran 1 test in 7.735s
2022-09-27T16:30:20.2564837Z 
2022-09-27T16:30:20.2565035Z OK
2022-09-27T16:30:20.2565354Z 
2022-09-27T16:30:20.2565660Z Generating XML reports...
2022-09-27T16:30:20.2607258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163012.xml
2022-09-27T16:30:21.2731628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:21.2732142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:21.2733252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:21.2733747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:21.7557030Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:21.7579328Z 
2022-09-27T16:30:21.7579621Z Running tests...
2022-09-27T16:30:21.7580068Z ----------------------------------------------------------------------
2022-09-27T16:30:21.7607736Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T16:30:21.7608049Z 
2022-09-27T16:30:21.7608330Z ----------------------------------------------------------------------
2022-09-27T16:30:21.7608673Z Ran 1 test in 0.003s
2022-09-27T16:30:21.7608842Z 
2022-09-27T16:30:21.7608952Z OK (skipped=1)
2022-09-27T16:30:21.7609114Z 
2022-09-27T16:30:21.7609223Z Generating XML reports...
2022-09-27T16:30:21.7643681Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163021.xml
2022-09-27T16:30:21.8700966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:21.8701495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:21.8703947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:21.8704448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:22.3620665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:22.3643204Z 
2022-09-27T16:30:22.3643461Z Running tests...
2022-09-27T16:30:22.3643901Z ----------------------------------------------------------------------
2022-09-27T16:30:22.3662417Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:22.3662776Z 
2022-09-27T16:30:22.3663067Z ----------------------------------------------------------------------
2022-09-27T16:30:22.3663409Z Ran 1 test in 0.002s
2022-09-27T16:30:22.3663576Z 
2022-09-27T16:30:22.3663679Z OK (skipped=1)
2022-09-27T16:30:22.3663841Z 
2022-09-27T16:30:22.3663967Z Generating XML reports...
2022-09-27T16:30:22.3699351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163022.xml
2022-09-27T16:30:22.4195918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:22.4196391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:22.4198576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:22.4199069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:22.9208214Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:22.9231059Z 
2022-09-27T16:30:22.9231478Z Running tests...
2022-09-27T16:30:22.9231980Z ----------------------------------------------------------------------
2022-09-27T16:30:22.9249948Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-09-27T16:30:22.9250752Z 
2022-09-27T16:30:22.9251187Z ----------------------------------------------------------------------
2022-09-27T16:30:22.9251537Z Ran 1 test in 0.002s
2022-09-27T16:30:22.9251707Z 
2022-09-27T16:30:22.9251817Z OK (skipped=1)
2022-09-27T16:30:22.9251975Z 
2022-09-27T16:30:22.9252084Z Generating XML reports...
2022-09-27T16:30:22.9287660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163022.xml
2022-09-27T16:30:23.6626288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:23.6626811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:23.6629470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:23.6630275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:24.2045286Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:24.2067624Z 
2022-09-27T16:30:24.2068061Z Running tests...
2022-09-27T16:30:24.2068589Z ----------------------------------------------------------------------
2022-09-27T16:30:24.2093842Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T16:30:24.2094922Z 
2022-09-27T16:30:24.2095229Z ----------------------------------------------------------------------
2022-09-27T16:30:24.2095584Z Ran 1 test in 0.003s
2022-09-27T16:30:24.2095751Z 
2022-09-27T16:30:24.2095880Z OK (skipped=1)
2022-09-27T16:30:24.2096022Z 
2022-09-27T16:30:24.2096148Z Generating XML reports...
2022-09-27T16:30:24.2129867Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163024.xml
2022-09-27T16:30:24.3621393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:24.3621901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:24.3624694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:24.3625188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:24.8643195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:24.8666605Z 
2022-09-27T16:30:24.8667201Z Running tests...
2022-09-27T16:30:24.8667756Z ----------------------------------------------------------------------
2022-09-27T16:30:24.8686304Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:24.8687470Z 
2022-09-27T16:30:24.8687954Z ----------------------------------------------------------------------
2022-09-27T16:30:24.8688684Z Ran 1 test in 0.002s
2022-09-27T16:30:24.8689037Z 
2022-09-27T16:30:24.8689227Z OK (skipped=1)
2022-09-27T16:30:24.8689397Z 
2022-09-27T16:30:24.8689524Z Generating XML reports...
2022-09-27T16:30:24.8702411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:24.8703166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:24.8706432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:24.8707377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:24.8725567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163024.xml
2022-09-27T16:30:25.4249732Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:25.4274541Z 
2022-09-27T16:30:25.4274704Z Running tests...
2022-09-27T16:30:25.4275173Z ----------------------------------------------------------------------
2022-09-27T16:30:26.1218458Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:26.1219070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:26.1220298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:26.1220775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:26.6246528Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:26.6268937Z 
2022-09-27T16:30:26.6269272Z Running tests...
2022-09-27T16:30:26.6269998Z ----------------------------------------------------------------------
2022-09-27T16:30:26.6290204Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:30:26.6290741Z 
2022-09-27T16:30:26.6291036Z ----------------------------------------------------------------------
2022-09-27T16:30:26.6291374Z Ran 1 test in 0.002s
2022-09-27T16:30:26.6291543Z 
2022-09-27T16:30:26.6291662Z OK (skipped=1)
2022-09-27T16:30:26.6291802Z 
2022-09-27T16:30:26.6291930Z Generating XML reports...
2022-09-27T16:30:26.6326262Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163026.xml
2022-09-27T16:30:26.8747205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:26.8747736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:26.8749975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:26.8750483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:26.9739803Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:26.9939247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87405
2022-09-27T16:30:26.9945084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87406
2022-09-27T16:30:27.3677918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:27.3702579Z 
2022-09-27T16:30:27.3702896Z Running tests...
2022-09-27T16:30:27.3703359Z ----------------------------------------------------------------------
2022-09-27T16:30:27.3721548Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:30:27.3722218Z 
2022-09-27T16:30:27.3722580Z ----------------------------------------------------------------------
2022-09-27T16:30:27.3722932Z Ran 1 test in 0.002s
2022-09-27T16:30:27.3723104Z 
2022-09-27T16:30:27.3723220Z OK (skipped=1)
2022-09-27T16:30:27.3723377Z 
2022-09-27T16:30:27.3723506Z Generating XML reports...
2022-09-27T16:30:27.3758630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163027.xml
2022-09-27T16:30:28.6851488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:28.6852023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:28.6853126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:28.6853913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:28.6902557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:28.6903055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:28.6906203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:28.6906677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:28.8005390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:28.8005891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:28.8008343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:28.8008839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:29.1896066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:29.1903180Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:29.1927976Z 
2022-09-27T16:30:29.1928202Z Running tests...
2022-09-27T16:30:29.1928648Z ----------------------------------------------------------------------
2022-09-27T16:30:29.1947705Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:30:29.1948019Z 
2022-09-27T16:30:29.1948288Z ----------------------------------------------------------------------
2022-09-27T16:30:29.1948644Z Ran 1 test in 0.002s
2022-09-27T16:30:29.1948808Z 
2022-09-27T16:30:29.1948920Z OK (skipped=1)
2022-09-27T16:30:29.1949081Z 
2022-09-27T16:30:29.1949210Z Generating XML reports...
2022-09-27T16:30:29.1984066Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163029.xml
2022-09-27T16:30:29.3093438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:29.3094227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:29.3112983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:29.3119434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:29.4106980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:29.4751375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:29.4751898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:29.4754260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:29.4754773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:29.8012368Z skip: Skipped due to small world size. (4.373s)
2022-09-27T16:30:29.8012629Z 
2022-09-27T16:30:29.8012984Z ----------------------------------------------------------------------
2022-09-27T16:30:29.8013329Z Ran 1 test in 4.374s
2022-09-27T16:30:29.8013494Z 
2022-09-27T16:30:29.8013605Z OK (skipped=1)
2022-09-27T16:30:29.8013762Z 
2022-09-27T16:30:29.8013893Z Generating XML reports...
2022-09-27T16:30:29.8055378Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163025.xml
2022-09-27T16:30:29.9907568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:29.9931764Z 
2022-09-27T16:30:29.9932241Z Running tests...
2022-09-27T16:30:29.9933055Z ----------------------------------------------------------------------
2022-09-27T16:30:29.9952258Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:29.9952770Z 
2022-09-27T16:30:29.9953118Z ----------------------------------------------------------------------
2022-09-27T16:30:29.9953467Z Ran 1 test in 0.002s
2022-09-27T16:30:29.9953637Z 
2022-09-27T16:30:29.9953741Z OK (skipped=1)
2022-09-27T16:30:29.9953899Z 
2022-09-27T16:30:29.9954048Z Generating XML reports...
2022-09-27T16:30:29.9989697Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163029.xml
2022-09-27T16:30:31.2323946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:31.2324937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:31.2326571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:31.2327542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:31.7154728Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:31.7177433Z 
2022-09-27T16:30:31.7177905Z Running tests...
2022-09-27T16:30:31.7178404Z ----------------------------------------------------------------------
2022-09-27T16:30:31.7200719Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:30:31.7201327Z 
2022-09-27T16:30:31.7201847Z ----------------------------------------------------------------------
2022-09-27T16:30:31.7202183Z Ran 1 test in 0.002s
2022-09-27T16:30:31.7202348Z 
2022-09-27T16:30:31.7202459Z OK (skipped=1)
2022-09-27T16:30:31.7202617Z 
2022-09-27T16:30:31.7202743Z Generating XML reports...
2022-09-27T16:30:31.7235559Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163031.xml
2022-09-27T16:30:31.9800674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:31.9801270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:31.9804001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:31.9804480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:32.0325269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:32.0325737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:32.0329921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:32.0330426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:32.4819075Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:32.4842723Z 
2022-09-27T16:30:32.4843036Z Running tests...
2022-09-27T16:30:32.4843488Z ----------------------------------------------------------------------
2022-09-27T16:30:32.4862772Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-09-27T16:30:32.4863084Z 
2022-09-27T16:30:32.4864065Z ----------------------------------------------------------------------
2022-09-27T16:30:32.4864760Z Ran 1 test in 0.002s
2022-09-27T16:30:32.4865078Z 
2022-09-27T16:30:32.4865286Z OK (skipped=1)
2022-09-27T16:30:32.4865572Z 
2022-09-27T16:30:32.4865817Z Generating XML reports...
2022-09-27T16:30:32.4900671Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163032.xml
2022-09-27T16:30:32.5190588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:32.5216437Z 
2022-09-27T16:30:32.5216929Z Running tests...
2022-09-27T16:30:32.5217444Z ----------------------------------------------------------------------
2022-09-27T16:30:32.5236958Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:30:32.5237315Z 
2022-09-27T16:30:32.5237619Z ----------------------------------------------------------------------
2022-09-27T16:30:32.5237935Z Ran 1 test in 0.002s
2022-09-27T16:30:32.5238106Z 
2022-09-27T16:30:32.5238216Z OK (skipped=1)
2022-09-27T16:30:32.5238373Z 
2022-09-27T16:30:32.5238505Z Generating XML reports...
2022-09-27T16:30:32.5273506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163032.xml
2022-09-27T16:30:33.6087963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:33.6088513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:33.6089789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:33.6090580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:34.1032028Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:34.1054510Z 
2022-09-27T16:30:34.1054826Z Running tests...
2022-09-27T16:30:34.1055269Z ----------------------------------------------------------------------
2022-09-27T16:30:34.1079450Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:30:34.1079859Z 
2022-09-27T16:30:34.1080566Z ----------------------------------------------------------------------
2022-09-27T16:30:34.1080899Z Ran 1 test in 0.003s
2022-09-27T16:30:34.1081081Z 
2022-09-27T16:30:34.1081192Z OK (skipped=1)
2022-09-27T16:30:34.1081348Z 
2022-09-27T16:30:34.1081474Z Generating XML reports...
2022-09-27T16:30:34.1115468Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163034.xml
2022-09-27T16:30:34.4101874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:34.4102379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:34.4104343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:34.4104823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:34.4418177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:34.4418673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:34.4422197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:34.4422696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:34.8907842Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:34.8931216Z 
2022-09-27T16:30:34.8931401Z Running tests...
2022-09-27T16:30:34.8931817Z ----------------------------------------------------------------------
2022-09-27T16:30:34.9385204Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:34.9410371Z 
2022-09-27T16:30:34.9410666Z Running tests...
2022-09-27T16:30:34.9411105Z ----------------------------------------------------------------------
2022-09-27T16:30:34.9431357Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:34.9431745Z 
2022-09-27T16:30:34.9432035Z ----------------------------------------------------------------------
2022-09-27T16:30:34.9432382Z Ran 1 test in 0.002s
2022-09-27T16:30:34.9432544Z 
2022-09-27T16:30:34.9432652Z OK (skipped=1)
2022-09-27T16:30:34.9432807Z 
2022-09-27T16:30:34.9432915Z Generating XML reports...
2022-09-27T16:30:34.9467815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163034.xml
2022-09-27T16:30:35.9979010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:35.9980008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:35.9984499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:35.9985959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:36.4247748Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:36.4442236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87807
2022-09-27T16:30:36.4447476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87808
2022-09-27T16:30:36.4931146Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:36.4954760Z 
2022-09-27T16:30:36.4955213Z Running tests...
2022-09-27T16:30:36.4955722Z ----------------------------------------------------------------------
2022-09-27T16:30:36.4975784Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:30:36.4976316Z 
2022-09-27T16:30:36.4976958Z ----------------------------------------------------------------------
2022-09-27T16:30:36.4977616Z Ran 1 test in 0.002s
2022-09-27T16:30:36.4977930Z 
2022-09-27T16:30:36.4978139Z OK (skipped=1)
2022-09-27T16:30:36.4978458Z 
2022-09-27T16:30:36.4978704Z Generating XML reports...
2022-09-27T16:30:36.5014955Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163036.xml
2022-09-27T16:30:36.9529120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:36.9529646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:36.9530770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:36.9531247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:37.5598637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:37.5643263Z 
2022-09-27T16:30:37.5643928Z Running tests...
2022-09-27T16:30:37.5644826Z ----------------------------------------------------------------------
2022-09-27T16:30:37.5683634Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.003s)
2022-09-27T16:30:37.5684340Z 
2022-09-27T16:30:37.5684934Z ----------------------------------------------------------------------
2022-09-27T16:30:37.5685572Z Ran 1 test in 0.004s
2022-09-27T16:30:37.5685884Z 
2022-09-27T16:30:37.5686102Z OK (skipped=1)
2022-09-27T16:30:37.5686425Z 
2022-09-27T16:30:37.5686664Z Generating XML reports...
2022-09-27T16:30:37.5749776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163037.xml
2022-09-27T16:30:38.1203833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:38.1204347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:38.1205675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:38.1206479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:38.1259934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:38.1260415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:38.1264154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:38.1264643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:38.5025571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:38.5026686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:38.5028022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:38.5029147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:38.6839563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:38.6903622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:38.6904422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:38.6940830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:38.6947093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:38.7966703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:38.9987897Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:39.0011641Z 
2022-09-27T16:30:39.0011818Z Running tests...
2022-09-27T16:30:39.0012254Z ----------------------------------------------------------------------
2022-09-27T16:30:39.1520755Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Skipped due to small world size. (4.258s)
2022-09-27T16:30:39.1521146Z 
2022-09-27T16:30:39.1521518Z ----------------------------------------------------------------------
2022-09-27T16:30:39.1521857Z Ran 1 test in 4.259s
2022-09-27T16:30:39.1522006Z 
2022-09-27T16:30:39.1522115Z OK (skipped=1)
2022-09-27T16:30:39.1522271Z 
2022-09-27T16:30:39.1522397Z Generating XML reports...
2022-09-27T16:30:39.1577606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163034.xml
2022-09-27T16:30:39.5243650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:39.5244169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:39.5246166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:39.5246666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:40.0329289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:40.0352046Z 
2022-09-27T16:30:40.0352194Z Running tests...
2022-09-27T16:30:40.0352783Z ----------------------------------------------------------------------
2022-09-27T16:30:40.0371138Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:30:40.0371494Z 
2022-09-27T16:30:40.0371779Z ----------------------------------------------------------------------
2022-09-27T16:30:40.0372111Z Ran 1 test in 0.002s
2022-09-27T16:30:40.0372261Z 
2022-09-27T16:30:40.0372378Z OK (skipped=1)
2022-09-27T16:30:40.0372535Z 
2022-09-27T16:30:40.0372667Z Generating XML reports...
2022-09-27T16:30:40.0408395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163040.xml
2022-09-27T16:30:40.6314324Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:40.6505553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87981
2022-09-27T16:30:40.6511873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87982
2022-09-27T16:30:41.3589942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:41.3591736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:41.3612740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:41.3614723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:41.9147584Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:41.9169907Z 
2022-09-27T16:30:41.9170213Z Running tests...
2022-09-27T16:30:41.9170663Z ----------------------------------------------------------------------
2022-09-27T16:30:41.9189190Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T16:30:41.9189554Z 
2022-09-27T16:30:41.9189879Z ----------------------------------------------------------------------
2022-09-27T16:30:41.9190225Z Ran 1 test in 0.002s
2022-09-27T16:30:41.9190391Z 
2022-09-27T16:30:41.9190505Z OK (skipped=1)
2022-09-27T16:30:41.9190648Z 
2022-09-27T16:30:41.9190779Z Generating XML reports...
2022-09-27T16:30:41.9225397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163041.xml
2022-09-27T16:30:42.1859401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:42.1859902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:42.1861339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:42.1861843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:42.3267560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:42.3268047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:42.3269034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:42.3269522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:42.4353991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:42.4354531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:42.4355837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:42.4356343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:42.6707691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:42.6729789Z 
2022-09-27T16:30:42.6730167Z Running tests...
2022-09-27T16:30:42.6730618Z ----------------------------------------------------------------------
2022-09-27T16:30:42.6748771Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:42.6749116Z 
2022-09-27T16:30:42.6749403Z ----------------------------------------------------------------------
2022-09-27T16:30:42.6749736Z Ran 1 test in 0.002s
2022-09-27T16:30:42.6749899Z 
2022-09-27T16:30:42.6749998Z OK (skipped=1)
2022-09-27T16:30:42.6750161Z 
2022-09-27T16:30:42.6750577Z Generating XML reports...
2022-09-27T16:30:42.6784065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163042.xml
2022-09-27T16:30:42.8299102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:42.9548981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:42.9550585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:42.9623381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:42.9630230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:42.9696055Z STAGE:2022-09-27 16:30:42 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:43.0562204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:43.0666183Z STAGE:2022-09-27 16:30:43 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:43.8232959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:43.8233486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:43.8235509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:43.8236006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:44.3409251Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:44.3432467Z 
2022-09-27T16:30:44.3432819Z Running tests...
2022-09-27T16:30:44.3457134Z ----------------------------------------------------------------------
2022-09-27T16:30:44.3458188Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T16:30:44.3458842Z 
2022-09-27T16:30:44.3459388Z ----------------------------------------------------------------------
2022-09-27T16:30:44.3460036Z Ran 1 test in 0.002s
2022-09-27T16:30:44.3460356Z 
2022-09-27T16:30:44.3460567Z OK (skipped=1)
2022-09-27T16:30:44.3460854Z 
2022-09-27T16:30:44.3461100Z Generating XML reports...
2022-09-27T16:30:44.3495747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163044.xml
2022-09-27T16:30:44.6362945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:44.6363922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:44.6365142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:44.6366093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:44.7307373Z [1664296244.730031] [28fd6574525b:87982:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:30:44.7308565Z [1664296244.730031] [28fd6574525b:87982:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:30:44.7714813Z [1664296244.771003] [28fd6574525b:87981:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:30:44.7715358Z [1664296244.771003] [28fd6574525b:87981:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:30:45.1384750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:45.1406618Z 
2022-09-27T16:30:45.1407035Z Running tests...
2022-09-27T16:30:45.1407832Z ----------------------------------------------------------------------
2022-09-27T16:30:45.1425663Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:45.1427188Z 
2022-09-27T16:30:45.1427704Z ----------------------------------------------------------------------
2022-09-27T16:30:45.1428039Z Ran 1 test in 0.002s
2022-09-27T16:30:45.1428212Z 
2022-09-27T16:30:45.1428325Z OK (skipped=1)
2022-09-27T16:30:45.1428485Z 
2022-09-27T16:30:45.1428614Z Generating XML reports...
2022-09-27T16:30:45.1462292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163045.xml
2022-09-27T16:30:45.1523136Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.1523564Z 
2022-09-27T16:30:45.2779198Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2788336Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2800154Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2810892Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2821057Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2840357Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2850713Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2851326Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2861519Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2867041Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2876203Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2877374Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2890046Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2891462Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2900468Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2901678Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2911332Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2917817Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2927436Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2928630Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2940953Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2942374Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2951237Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2952469Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2962755Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2968832Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2978043Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2979254Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.2991556Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.2992987Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.3001680Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.3002890Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.3012942Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.3019402Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.3028621Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.3029832Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.3042298Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.3043693Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.3052472Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.3053703Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.3063414Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.3069943Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:45.3078983Z STAGE:2022-09-27 16:30:45 87981:87981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:45.3080499Z STAGE:2022-09-27 16:30:45 87982:87982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:46.4114039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:46.4114561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:46.4116833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:46.4117340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:46.4639917Z ok (7.463s)
2022-09-27T16:30:46.4640101Z 
2022-09-27T16:30:46.4640674Z ----------------------------------------------------------------------
2022-09-27T16:30:46.4641017Z Ran 1 test in 7.463s
2022-09-27T16:30:46.4641195Z 
2022-09-27T16:30:46.4641292Z OK
2022-09-27T16:30:46.4641431Z 
2022-09-27T16:30:46.4641565Z Generating XML reports...
2022-09-27T16:30:46.4676581Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163038.xml
2022-09-27T16:30:46.9272017Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:46.9295618Z 
2022-09-27T16:30:46.9296049Z Running tests...
2022-09-27T16:30:46.9296542Z ----------------------------------------------------------------------
2022-09-27T16:30:46.9315568Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-09-27T16:30:46.9315921Z 
2022-09-27T16:30:46.9316521Z ----------------------------------------------------------------------
2022-09-27T16:30:46.9316883Z Ran 1 test in 0.002s
2022-09-27T16:30:46.9317035Z 
2022-09-27T16:30:46.9317147Z OK (skipped=1)
2022-09-27T16:30:46.9317315Z 
2022-09-27T16:30:46.9317445Z Generating XML reports...
2022-09-27T16:30:46.9351779Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163046.xml
2022-09-27T16:30:47.0554992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:47.0555519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:47.0557898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:47.0558401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:47.5682054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:47.5723176Z 
2022-09-27T16:30:47.5726120Z Running tests...
2022-09-27T16:30:47.5727473Z ----------------------------------------------------------------------
2022-09-27T16:30:47.5758446Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.003s)
2022-09-27T16:30:47.5759222Z 
2022-09-27T16:30:47.5761182Z ----------------------------------------------------------------------
2022-09-27T16:30:47.5762472Z Ran 1 test in 0.003s
2022-09-27T16:30:47.5763059Z 
2022-09-27T16:30:47.5763386Z OK (skipped=1)
2022-09-27T16:30:47.5764959Z 
2022-09-27T16:30:47.5765250Z Generating XML reports...
2022-09-27T16:30:47.5818661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163047.xml
2022-09-27T16:30:48.5623889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:48.5624409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:48.5627142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:48.5627656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:49.0343487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:49.0343995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:49.0346482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:49.0346962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:49.0977936Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:49.1001177Z 
2022-09-27T16:30:49.1001434Z Running tests...
2022-09-27T16:30:49.1001882Z ----------------------------------------------------------------------
2022-09-27T16:30:49.1022005Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and Nccl backend supports CUDA allReduce (0.002s)
2022-09-27T16:30:49.1022407Z 
2022-09-27T16:30:49.1022919Z ----------------------------------------------------------------------
2022-09-27T16:30:49.1023380Z Ran 1 test in 0.002s
2022-09-27T16:30:49.1023549Z 
2022-09-27T16:30:49.1023661Z OK (skipped=1)
2022-09-27T16:30:49.1023819Z 
2022-09-27T16:30:49.1023929Z Generating XML reports...
2022-09-27T16:30:49.1056862Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163049.xml
2022-09-27T16:30:49.5388862Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:49.5413003Z 
2022-09-27T16:30:49.5413284Z Running tests...
2022-09-27T16:30:49.5413721Z ----------------------------------------------------------------------
2022-09-27T16:30:49.5436513Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-09-27T16:30:49.5437046Z 
2022-09-27T16:30:49.5437418Z ----------------------------------------------------------------------
2022-09-27T16:30:49.5437764Z Ran 1 test in 0.002s
2022-09-27T16:30:49.5437931Z 
2022-09-27T16:30:49.5438042Z OK (skipped=1)
2022-09-27T16:30:49.5438199Z 
2022-09-27T16:30:49.5438308Z Generating XML reports...
2022-09-27T16:30:49.5473024Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163049.xml
2022-09-27T16:30:49.5679154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:49.5679644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:49.5682564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:49.5683040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:50.1136204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:50.1157656Z 
2022-09-27T16:30:50.1157969Z Running tests...
2022-09-27T16:30:50.1158425Z ----------------------------------------------------------------------
2022-09-27T16:30:50.1177623Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:50.1178017Z 
2022-09-27T16:30:50.1178370Z ----------------------------------------------------------------------
2022-09-27T16:30:50.1178983Z Ran 1 test in 0.002s
2022-09-27T16:30:50.1179158Z 
2022-09-27T16:30:50.1179272Z OK (skipped=1)
2022-09-27T16:30:50.1179430Z 
2022-09-27T16:30:50.1179564Z Generating XML reports...
2022-09-27T16:30:50.1213015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163050.xml
2022-09-27T16:30:51.0942603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:51.0944417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:51.0948710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:51.0949857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:51.5450133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:51.5450629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:51.5452863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:51.5453370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:51.6389668Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:30:51.6413400Z 
2022-09-27T16:30:51.6413860Z Running tests...
2022-09-27T16:30:51.6414381Z ----------------------------------------------------------------------
2022-09-27T16:30:52.0310182Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:52.0332468Z 
2022-09-27T16:30:52.0332808Z Running tests...
2022-09-27T16:30:52.0333260Z ----------------------------------------------------------------------
2022-09-27T16:30:52.0355572Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-09-27T16:30:52.0355890Z 
2022-09-27T16:30:52.0356176Z ----------------------------------------------------------------------
2022-09-27T16:30:52.0356489Z Ran 1 test in 0.002s
2022-09-27T16:30:52.0356658Z 
2022-09-27T16:30:52.0356994Z OK (skipped=1)
2022-09-27T16:30:52.0357175Z 
2022-09-27T16:30:52.0357309Z Generating XML reports...
2022-09-27T16:30:52.0391437Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163052.xml
2022-09-27T16:30:52.0709779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:52.0710279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:52.0712280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:52.0712777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:52.5749700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:52.5772079Z 
2022-09-27T16:30:52.5772338Z Running tests...
2022-09-27T16:30:52.5772773Z ----------------------------------------------------------------------
2022-09-27T16:30:52.5791177Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-09-27T16:30:52.5791836Z 
2022-09-27T16:30:52.5792124Z ----------------------------------------------------------------------
2022-09-27T16:30:52.5792453Z Ran 1 test in 0.002s
2022-09-27T16:30:52.5792600Z 
2022-09-27T16:30:52.5792711Z OK (skipped=1)
2022-09-27T16:30:52.5792868Z 
2022-09-27T16:30:52.5792996Z Generating XML reports...
2022-09-27T16:30:52.5826820Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163052.xml
2022-09-27T16:30:53.2774238Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:53.2973374Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88460
2022-09-27T16:30:53.2979721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88461
2022-09-27T16:30:54.0721715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:54.0722648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:54.0723749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:54.0724498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:54.5519841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:30:54.5542959Z 
2022-09-27T16:30:54.5543339Z Running tests...
2022-09-27T16:30:54.5543815Z ----------------------------------------------------------------------
2022-09-27T16:30:54.6194400Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:54.6195358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:54.6197131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:54.6197626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:54.9508833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:54.9509822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:54.9510988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:54.9511930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:54.9519183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:54.9520119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:54.9523113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:54.9524085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:55.1574753Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:55.1598303Z 
2022-09-27T16:30:55.1598758Z Running tests...
2022-09-27T16:30:55.1599284Z ----------------------------------------------------------------------
2022-09-27T16:30:55.1618201Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-09-27T16:30:55.1618563Z 
2022-09-27T16:30:55.1618856Z ----------------------------------------------------------------------
2022-09-27T16:30:55.1619171Z Ran 1 test in 0.002s
2022-09-27T16:30:55.1619342Z 
2022-09-27T16:30:55.1619459Z OK (skipped=1)
2022-09-27T16:30:55.1619621Z 
2022-09-27T16:30:55.1619767Z Generating XML reports...
2022-09-27T16:30:55.1654738Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163055.xml
2022-09-27T16:30:55.4811862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:55.4836037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:55.4837050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:55.4913370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:55.4919901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:55.4923568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:30:55.5851437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:55.5853837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:30:55.5854940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:30:55.5920591Z STAGE:2022-09-27 16:30:55 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:55.5938127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:30:55.5999079Z STAGE:2022-09-27 16:30:55 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:56.1616056Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:56.1938696Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88603
2022-09-27T16:30:56.2027041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88604
2022-09-27T16:30:57.1933866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:57.1934391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:57.1936525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:57.1937027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:57.3109199Z [1664296257.310420] [28fd6574525b:88461:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:30:57.3109718Z [1664296257.310420] [28fd6574525b:88461:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:30:57.3142534Z [1664296257.313953] [28fd6574525b:88460:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:30:57.3143265Z [1664296257.313953] [28fd6574525b:88460:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:30:57.6841733Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.6842207Z 
2022-09-27T16:30:57.7060377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:30:57.7084589Z 
2022-09-27T16:30:57.7084913Z Running tests...
2022-09-27T16:30:57.7085372Z ----------------------------------------------------------------------
2022-09-27T16:30:57.8096273Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8105414Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8109785Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8116554Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8120229Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8140023Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8149294Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8149898Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8160110Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8163044Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8173164Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8173619Z 
2022-09-27T16:30:57.8183003Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8185788Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8195181Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8195800Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8206234Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8209133Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8218271Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8218878Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8228814Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8231991Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8241428Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8242029Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8252676Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8255385Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8264737Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8265151Z 
2022-09-27T16:30:57.8274908Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8277918Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8287363Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8287968Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8298362Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8300863Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8310424Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8310953Z 
2022-09-27T16:30:57.8320581Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8324803Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8334013Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8335045Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8346992Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8347672Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:30:57.8357196Z STAGE:2022-09-27 16:30:57 88461:88461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8357804Z STAGE:2022-09-27 16:30:57 88460:88460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:30:57.8970935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:57.8971643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:57.8973210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:57.8973768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:57.9339611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:30:57.9340550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:30:57.9341800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:30:57.9342356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:30:58.4175459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:30:58.4656662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:30:58.4657494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:58.4681735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:30:58.4689079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:30:58.5669879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:30:58.5821694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:30:58.5822620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:30:58.5823444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:30:58.5824131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:30:58.5825526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:30:58.5926187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:30:58.5926882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:30:58.5928052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:30:59.0109244Z ok (7.369s)
2022-09-27T16:30:59.0109589Z 
2022-09-27T16:30:59.0110369Z ----------------------------------------------------------------------
2022-09-27T16:30:59.0110940Z Ran 1 test in 7.369s
2022-09-27T16:30:59.0111109Z 
2022-09-27T16:30:59.0111210Z OK
2022-09-27T16:30:59.0111328Z 
2022-09-27T16:30:59.0111465Z Generating XML reports...
2022-09-27T16:30:59.0147385Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163051.xml
2022-09-27T16:30:59.1099737Z ok (4.555s)
2022-09-27T16:30:59.1100041Z 
2022-09-27T16:30:59.1100643Z ----------------------------------------------------------------------
2022-09-27T16:30:59.1100989Z Ran 1 test in 4.556s
2022-09-27T16:30:59.1101140Z 
2022-09-27T16:30:59.1101240Z OK
2022-09-27T16:30:59.1101379Z 
2022-09-27T16:30:59.1101533Z Generating XML reports...
2022-09-27T16:30:59.1136722Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163054.xml
2022-09-27T16:30:59.3527881Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:30:59.3736445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88693
2022-09-27T16:30:59.3742373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88694
2022-09-27T16:31:01.1825964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:01.1826462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:01.1828977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:01.1829478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:01.1972047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:01.1972519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:01.1975933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:01.1976429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:01.2150269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:01.2150743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:01.2154092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:01.2154582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:01.3683994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:01.3684504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:01.3685988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:01.3686486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:01.7034826Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:01.7057774Z 
2022-09-27T16:31:01.7058096Z Running tests...
2022-09-27T16:31:01.7058518Z ----------------------------------------------------------------------
2022-09-27T16:31:01.7508049Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:01.7508677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:01.7509446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:01.7510133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:01.7516699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:01.7517692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:01.8738038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:31:01.8758336Z 
2022-09-27T16:31:01.8758550Z Running tests...
2022-09-27T16:31:01.8758967Z ----------------------------------------------------------------------
2022-09-27T16:31:03.4846393Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:03.5043502Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88839
2022-09-27T16:31:03.5049395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88840
2022-09-27T16:31:03.5384601Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:03.5590598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88842
2022-09-27T16:31:03.5595019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88843
2022-09-27T16:31:04.2911144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:31:04.2919868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:31:04.2921217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:31:04.3013548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:31:05.0863503Z ok (7.377s)
2022-09-27T16:31:05.0863723Z 
2022-09-27T16:31:05.0864132Z ----------------------------------------------------------------------
2022-09-27T16:31:05.0864481Z Ran 1 test in 7.378s
2022-09-27T16:31:05.0864665Z 
2022-09-27T16:31:05.0864761Z OK
2022-09-27T16:31:05.0864898Z 
2022-09-27T16:31:05.0865021Z Generating XML reports...
2022-09-27T16:31:05.0903395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163057.xml
2022-09-27T16:31:05.1824321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:05.1825196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:05.1826163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:05.1826980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:05.2217662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:05.2218946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:05.2220467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:05.2221632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:05.2497068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:05.2497956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:05.2499021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:05.2499803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:05.2983997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:05.2985248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:05.2986747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:05.2988299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:05.7265883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:05.7433181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:05.7434061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:05.7468979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:05.7475645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:05.7638451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:05.8063752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:05.8064789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:05.8146529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:05.8153814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:05.8446019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:05.9077997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:06.2130370Z skip: Skipped due to small world size. (4.506s)
2022-09-27T16:31:06.2130950Z 
2022-09-27T16:31:06.2131789Z ----------------------------------------------------------------------
2022-09-27T16:31:06.2132561Z Ran 1 test in 4.507s
2022-09-27T16:31:06.2132945Z 
2022-09-27T16:31:06.2133195Z OK (skipped=1)
2022-09-27T16:31:06.2133589Z 
2022-09-27T16:31:06.2133891Z Generating XML reports...
2022-09-27T16:31:06.2193892Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163101.xml
2022-09-27T16:31:07.2925976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:07.2926492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:07.2927824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:07.2928320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:07.8752859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:31:07.8777835Z 
2022-09-27T16:31:07.8778119Z Running tests...
2022-09-27T16:31:07.8778582Z ----------------------------------------------------------------------
2022-09-27T16:31:07.9695591Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... ok (6.093s)
2022-09-27T16:31:07.9695884Z 
2022-09-27T16:31:07.9696295Z ----------------------------------------------------------------------
2022-09-27T16:31:07.9696637Z Ran 1 test in 6.093s
2022-09-27T16:31:07.9696804Z 
2022-09-27T16:31:07.9696901Z OK
2022-09-27T16:31:07.9697040Z 
2022-09-27T16:31:07.9697150Z Generating XML reports...
2022-09-27T16:31:07.9730507Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163101.xml
2022-09-27T16:31:08.5156831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:08.5157343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:08.5159406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:08.5160402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:09.0141812Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:09.0163901Z 
2022-09-27T16:31:09.0164436Z Running tests...
2022-09-27T16:31:09.0165314Z ----------------------------------------------------------------------
2022-09-27T16:31:09.5152788Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:09.5356025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89103
2022-09-27T16:31:09.5362425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89104
2022-09-27T16:31:10.0457931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:10.0458672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:10.0460118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:10.0460855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:10.6024396Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:31:10.6048322Z 
2022-09-27T16:31:10.6048839Z Running tests...
2022-09-27T16:31:10.6049339Z ----------------------------------------------------------------------
2022-09-27T16:31:10.6234540Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:10.6430547Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89169
2022-09-27T16:31:10.6436720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89170
2022-09-27T16:31:11.1380948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:11.1381495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:11.1383313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:11.1383793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:11.1606748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:11.1607255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:11.1609622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:11.1610371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:11.7572330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:11.7572906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:11.7573716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:11.7574407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:11.7580268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:11.7582001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:12.1432363Z skip: Need at least 3 CUDA devices (4.265s)
2022-09-27T16:31:12.1432592Z 
2022-09-27T16:31:12.1433198Z ----------------------------------------------------------------------
2022-09-27T16:31:12.1433554Z Ran 1 test in 4.265s
2022-09-27T16:31:12.1433727Z 
2022-09-27T16:31:12.1433846Z OK (skipped=1)
2022-09-27T16:31:12.1434007Z 
2022-09-27T16:31:12.1434405Z Generating XML reports...
2022-09-27T16:31:12.1468818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163107.xml
2022-09-27T16:31:12.2433189Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:12.2636618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89245
2022-09-27T16:31:12.2642387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89246
2022-09-27T16:31:12.3269851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:12.3270488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:12.3272271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:12.3272743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:12.3659812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:12.3660331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:12.3661877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:12.3662350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:12.8366870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:12.8793118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:12.8794006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:12.8873865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:12.8882849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:12.9831415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:13.9738117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:13.9738629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:13.9739859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:13.9740354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:14.0028361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:14.0029054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:14.0030103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:14.0030607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:14.3687642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:14.3688139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:14.3690573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:14.3691079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:14.3753305Z [1664296274.375042] [28fd6574525b:89169:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:14.3753808Z [1664296274.375042] [28fd6574525b:89169:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:14.3781851Z [1664296274.377914] [28fd6574525b:89170:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:14.3782590Z [1664296274.377914] [28fd6574525b:89170:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:14.4997753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:14.5346022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:14.5346814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:14.5403715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:14.5410728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:14.6360564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:14.8845525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:31:14.8870398Z 
2022-09-27T16:31:14.8870816Z Running tests...
2022-09-27T16:31:14.8871623Z ----------------------------------------------------------------------
2022-09-27T16:31:14.8891765Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-09-27T16:31:14.8892339Z 
2022-09-27T16:31:14.8892851Z ----------------------------------------------------------------------
2022-09-27T16:31:14.8893423Z Ran 1 test in 0.002s
2022-09-27T16:31:14.8893693Z 
2022-09-27T16:31:14.8893891Z OK (skipped=1)
2022-09-27T16:31:14.8894148Z 
2022-09-27T16:31:14.8894368Z Generating XML reports...
2022-09-27T16:31:14.8930759Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163114.xml
2022-09-27T16:31:15.0710596Z skip: Skipped due to small world size. (4.466s)
2022-09-27T16:31:15.0710897Z 
2022-09-27T16:31:15.0711282Z ----------------------------------------------------------------------
2022-09-27T16:31:15.0711631Z Ran 1 test in 4.466s
2022-09-27T16:31:15.0711798Z 
2022-09-27T16:31:15.0711891Z OK (skipped=1)
2022-09-27T16:31:15.0712052Z 
2022-09-27T16:31:15.0712180Z Generating XML reports...
2022-09-27T16:31:15.0748146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163110.xml
2022-09-27T16:31:15.2545311Z ok (6.238s)
2022-09-27T16:31:15.2545511Z 
2022-09-27T16:31:15.2545894Z ----------------------------------------------------------------------
2022-09-27T16:31:15.2546234Z Ran 1 test in 6.238s
2022-09-27T16:31:15.2546405Z 
2022-09-27T16:31:15.2546508Z OK
2022-09-27T16:31:15.2546961Z 
2022-09-27T16:31:15.2547099Z Generating XML reports...
2022-09-27T16:31:15.2581718Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163109.xml
2022-09-27T16:31:16.9965619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:16.9966354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:16.9968134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:16.9968619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:17.2544182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:17.2544750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:17.2546509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:17.2547014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:17.4373998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:17.4374524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:17.4375551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:17.4376041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:17.5008635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:31:17.5032685Z 
2022-09-27T16:31:17.5033031Z Running tests...
2022-09-27T16:31:17.5033473Z ----------------------------------------------------------------------
2022-09-27T16:31:17.7385761Z   test_barrier (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:31:17.7407937Z 
2022-09-27T16:31:17.7408242Z Running tests...
2022-09-27T16:31:17.7408694Z ----------------------------------------------------------------------
2022-09-27T16:31:17.9183951Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:17.9206802Z 
2022-09-27T16:31:17.9207224Z Running tests...
2022-09-27T16:31:17.9207731Z ----------------------------------------------------------------------
2022-09-27T16:31:17.9226443Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:31:17.9226797Z 
2022-09-27T16:31:17.9227073Z ----------------------------------------------------------------------
2022-09-27T16:31:17.9227416Z Ran 1 test in 0.002s
2022-09-27T16:31:17.9227602Z 
2022-09-27T16:31:17.9227714Z OK (skipped=1)
2022-09-27T16:31:17.9227873Z 
2022-09-27T16:31:17.9227983Z Generating XML reports...
2022-09-27T16:31:17.9262205Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163117.xml
2022-09-27T16:31:19.1729024Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:19.1927071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89499
2022-09-27T16:31:19.1932876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89500
2022-09-27T16:31:19.3717748Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:19.4007290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89502
2022-09-27T16:31:19.4012380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89503
2022-09-27T16:31:19.8852876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:19.8856294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:19.8858333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:19.8859447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:20.5589977Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:20.5612446Z 
2022-09-27T16:31:20.5612817Z Running tests...
2022-09-27T16:31:20.5631142Z ----------------------------------------------------------------------
2022-09-27T16:31:20.5631899Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:31:20.5632288Z 
2022-09-27T16:31:20.5632559Z ----------------------------------------------------------------------
2022-09-27T16:31:20.5632883Z Ran 1 test in 0.002s
2022-09-27T16:31:20.5633068Z 
2022-09-27T16:31:20.5633184Z OK (skipped=1)
2022-09-27T16:31:20.5633345Z 
2022-09-27T16:31:20.5633474Z Generating XML reports...
2022-09-27T16:31:20.5667758Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163120.xml
2022-09-27T16:31:20.9650838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:20.9651351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:20.9653641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:20.9654149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:20.9665897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:20.9666365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:20.9669844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:20.9670347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:21.1390883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:21.1391409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:21.1392848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:21.1393334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:21.1835449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:21.1835922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:21.1838666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:21.1839160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:21.5057279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:21.5059068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:21.5060215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:21.5061383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:21.5248494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:21.6073256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:21.6440697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:21.7427849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:21.7428704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:21.7454022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:21.7461211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:21.8437175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:22.4942770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:22.4943594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:22.4944740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:22.4945256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:22.9019940Z ok (5.398s)
2022-09-27T16:31:22.9020177Z 
2022-09-27T16:31:22.9020578Z ----------------------------------------------------------------------
2022-09-27T16:31:22.9020908Z Ran 1 test in 5.398s
2022-09-27T16:31:22.9021077Z 
2022-09-27T16:31:22.9021174Z OK
2022-09-27T16:31:22.9021316Z 
2022-09-27T16:31:22.9021458Z Generating XML reports...
2022-09-27T16:31:22.9056594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163117.xml
2022-09-27T16:31:23.0201535Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:23.0225420Z 
2022-09-27T16:31:23.0225902Z Running tests...
2022-09-27T16:31:23.0226440Z ----------------------------------------------------------------------
2022-09-27T16:31:23.0244025Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:31:23.0245230Z 
2022-09-27T16:31:23.0245545Z ----------------------------------------------------------------------
2022-09-27T16:31:23.0246157Z Ran 1 test in 0.002s
2022-09-27T16:31:23.0246331Z 
2022-09-27T16:31:23.0246442Z OK (skipped=1)
2022-09-27T16:31:23.0246581Z 
2022-09-27T16:31:23.0246712Z Generating XML reports...
2022-09-27T16:31:23.0281078Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163123.xml
2022-09-27T16:31:23.5103290Z ok (5.769s)
2022-09-27T16:31:23.5103629Z 
2022-09-27T16:31:23.5104210Z ----------------------------------------------------------------------
2022-09-27T16:31:23.5104559Z Ran 1 test in 5.769s
2022-09-27T16:31:23.5104724Z 
2022-09-27T16:31:23.5104817Z OK
2022-09-27T16:31:23.5104952Z 
2022-09-27T16:31:23.5105086Z Generating XML reports...
2022-09-27T16:31:23.5147028Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163117.xml
2022-09-27T16:31:25.0107905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:25.0108434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:25.0111600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:25.0112099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:25.0431771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:25.0432260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:25.0435730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:25.0436264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:25.5006240Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:25.5028051Z 
2022-09-27T16:31:25.5028318Z Running tests...
2022-09-27T16:31:25.5028760Z ----------------------------------------------------------------------
2022-09-27T16:31:25.5467045Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:31:25.5491913Z 
2022-09-27T16:31:25.5492301Z Running tests...
2022-09-27T16:31:25.5492745Z ----------------------------------------------------------------------
2022-09-27T16:31:25.5493356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:25.5493841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:25.5499079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:25.5499837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:26.0501865Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:31:26.0525969Z 
2022-09-27T16:31:26.0526427Z Running tests...
2022-09-27T16:31:26.0526926Z ----------------------------------------------------------------------
2022-09-27T16:31:27.1348182Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:27.1547882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89784
2022-09-27T16:31:27.1553810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89785
2022-09-27T16:31:27.1786498Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:27.1992361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89787
2022-09-27T16:31:27.1997098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89788
2022-09-27T16:31:27.6292474Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:27.6489293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89790
2022-09-27T16:31:27.6494720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89791
2022-09-27T16:31:28.9546395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:28.9546916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:28.9547514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:28.9548019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:28.9837861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:28.9838517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:28.9841156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:28.9841643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:28.9912165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:28.9912613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:28.9916702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:28.9917462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:29.0191101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:29.0191563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:29.0195161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:29.0195649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:29.4610860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:29.4685736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:29.4686205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:29.4689854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:29.4690361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:29.5155473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:29.5156374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:29.5158781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:29.5159250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:29.5161538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:29.5162241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:29.5218323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:29.5225401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:29.5703487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:29.5704046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:29.5704824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:29.5705524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:29.5812646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:29.6176726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:29.6718059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:30.0062156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:30.0436527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:30.0437357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:30.0469141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:30.0476742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:30.1449778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:30.6574110Z ok (4.604s)
2022-09-27T16:31:30.6574368Z 
2022-09-27T16:31:30.6574751Z ----------------------------------------------------------------------
2022-09-27T16:31:30.6575089Z Ran 1 test in 4.605s
2022-09-27T16:31:30.6575541Z 
2022-09-27T16:31:30.6575649Z OK
2022-09-27T16:31:30.6575777Z 
2022-09-27T16:31:30.6575916Z Generating XML reports...
2022-09-27T16:31:30.6611824Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163126.xml
2022-09-27T16:31:30.9195759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_aifvw0d
2022-09-27T16:31:30.9196380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_aifvw0d/_remote_module_non_scriptable.py
2022-09-27T16:31:31.0121427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa86xmw23
2022-09-27T16:31:31.0122036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa86xmw23/_remote_module_non_scriptable.py
2022-09-27T16:31:31.0469831Z [1664296291.046471] [28fd6574525b:89784:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:31.0470403Z [1664296291.046471] [28fd6574525b:89784:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:31.0493187Z [1664296291.049003] [28fd6574525b:89785:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:31.0493898Z [1664296291.049003] [28fd6574525b:89785:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:31.8486683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:31.8489698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:32.5111798Z ok (6.962s)
2022-09-27T16:31:32.5112046Z 
2022-09-27T16:31:32.5112450Z ----------------------------------------------------------------------
2022-09-27T16:31:32.5112796Z Ran 1 test in 6.962s
2022-09-27T16:31:32.5112965Z 
2022-09-27T16:31:32.5113060Z OK
2022-09-27T16:31:32.5113178Z 
2022-09-27T16:31:32.5114484Z Generating XML reports...
2022-09-27T16:31:32.5149046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163125.xml
2022-09-27T16:31:32.5679985Z ok (7.065s)
2022-09-27T16:31:32.5680172Z 
2022-09-27T16:31:32.5680835Z ----------------------------------------------------------------------
2022-09-27T16:31:32.5681196Z Ran 1 test in 7.065s
2022-09-27T16:31:32.5681358Z 
2022-09-27T16:31:32.5681457Z OK
2022-09-27T16:31:32.5681591Z 
2022-09-27T16:31:32.5681725Z Generating XML reports...
2022-09-27T16:31:32.5716798Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163125.xml
2022-09-27T16:31:32.9184609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:32.9185118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:32.9188219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:32.9188695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:33.4328410Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:31:33.4350288Z 
2022-09-27T16:31:33.4350570Z Running tests...
2022-09-27T16:31:33.4351012Z ----------------------------------------------------------------------
2022-09-27T16:31:34.6500875Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:34.6501479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:34.6503950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:34.6504445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:34.6937411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:34.6937915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:34.6940966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:34.6941459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:35.0778604Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:35.0969260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90118
2022-09-27T16:31:35.0976231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90119
2022-09-27T16:31:35.1760821Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:31:35.1784341Z 
2022-09-27T16:31:35.1784793Z Running tests...
2022-09-27T16:31:35.1785328Z ----------------------------------------------------------------------
2022-09-27T16:31:35.1990283Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:35.2014681Z 
2022-09-27T16:31:35.2014910Z Running tests...
2022-09-27T16:31:35.2015359Z ----------------------------------------------------------------------
2022-09-27T16:31:36.7488887Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:36.7489497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:36.7492150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:36.7492656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:36.7573169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:36.7573646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:36.7576252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:36.7576748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:36.8727424Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:36.8929186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90187
2022-09-27T16:31:36.8935032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90188
2022-09-27T16:31:36.8944690Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:36.9146340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90190
2022-09-27T16:31:36.9152820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90191
2022-09-27T16:31:37.2712288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:37.2719228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:37.2720059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:37.2813115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:37.2819458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:37.3735471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:38.6142753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:38.6143278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:38.6144413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:38.6144915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:38.6202507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:38.6202978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:38.6206478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:38.6206951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:38.6231988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:38.6232452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:38.6235750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:38.6236223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:38.6417668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:38.6418156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:38.6418948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:38.6419408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:39.1564699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:39.1566703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:39.1567520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:39.1666083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:39.1672248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:39.2138809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:39.2139348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:39.2140120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:39.2140818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:39.2249757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:39.2580667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:39.3153576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:39.3362374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:31:39.3363151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:31:39.3363990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:31:39.3364703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:31:39.4071815Z ok (5.972s)
2022-09-27T16:31:39.4072029Z 
2022-09-27T16:31:39.4072394Z ----------------------------------------------------------------------
2022-09-27T16:31:39.4072976Z Ran 1 test in 5.972s
2022-09-27T16:31:39.4073169Z 
2022-09-27T16:31:39.4073275Z OK
2022-09-27T16:31:39.4073407Z 
2022-09-27T16:31:39.4073542Z Generating XML reports...
2022-09-27T16:31:39.4110238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163133.xml
2022-09-27T16:31:40.5701701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydjbh5yy
2022-09-27T16:31:40.5703181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydjbh5yy/_remote_module_non_scriptable.py
2022-09-27T16:31:40.6030347Z ok (5.424s)
2022-09-27T16:31:40.6030789Z 
2022-09-27T16:31:40.6031497Z ----------------------------------------------------------------------
2022-09-27T16:31:40.6032175Z Ran 1 test in 5.424s
2022-09-27T16:31:40.6032507Z 
2022-09-27T16:31:40.6032703Z OK
2022-09-27T16:31:40.6032960Z 
2022-09-27T16:31:40.6033236Z Generating XML reports...
2022-09-27T16:31:40.6100083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163135.xml
2022-09-27T16:31:40.6911489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5uhneln
2022-09-27T16:31:40.6914140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5uhneln/_remote_module_non_scriptable.py
2022-09-27T16:31:40.7182054Z [1664296300.717736] [28fd6574525b:90190:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:40.7182569Z [1664296300.717736] [28fd6574525b:90190:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:40.7207538Z [1664296300.720458] [28fd6574525b:90191:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:40.7208015Z [1664296300.720458] [28fd6574525b:90191:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:41.5158047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:41.5158575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:41.5165837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:41.5166338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:41.6010454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:41.6010973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:41.6012259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:41.6012749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:42.1217900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:31:42.1239921Z 
2022-09-27T16:31:42.1240166Z Running tests...
2022-09-27T16:31:42.1240844Z ----------------------------------------------------------------------
2022-09-27T16:31:42.1272688Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... ok (6.926s)
2022-09-27T16:31:42.1272984Z 
2022-09-27T16:31:42.1273274Z ----------------------------------------------------------------------
2022-09-27T16:31:42.1273624Z Ran 1 test in 6.926s
2022-09-27T16:31:42.1273775Z 
2022-09-27T16:31:42.1273869Z OK
2022-09-27T16:31:42.1274009Z 
2022-09-27T16:31:42.1274132Z Generating XML reports...
2022-09-27T16:31:42.1310743Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163135.xml
2022-09-27T16:31:42.7863699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:42.7864255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:42.7866707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:42.7867228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:43.2911652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:31:43.2939088Z 
2022-09-27T16:31:43.2939929Z Running tests...
2022-09-27T16:31:43.2940859Z ----------------------------------------------------------------------
2022-09-27T16:31:43.7334770Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:43.7534214Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90469
2022-09-27T16:31:43.7539641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90470
2022-09-27T16:31:44.1765111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:44.1766140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:44.1768382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:44.1769709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:44.7408682Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:44.7431505Z 
2022-09-27T16:31:44.7431776Z Running tests...
2022-09-27T16:31:44.7432225Z ----------------------------------------------------------------------
2022-09-27T16:31:44.9244677Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:44.9450695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90536
2022-09-27T16:31:44.9456312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90537
2022-09-27T16:31:45.4029544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:45.4030587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:45.4031818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:45.4032767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:45.4792474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:45.4793509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:45.4794745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:45.4795641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:45.9368397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:46.0052453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:46.0053319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:46.0077148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:46.0084824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:46.1065445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:46.3965801Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:46.4172069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90611
2022-09-27T16:31:46.4176454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90612
2022-09-27T16:31:46.6009753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:46.6010363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:46.6011342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:46.6012100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:46.6079031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:46.6079537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:46.6083011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:46.6083658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:47.2159548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:47.2160817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:47.2162358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:47.2163602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:47.2268827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:47.3172494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:47.7535352Z skip: Skipped due to small world size. (4.459s)
2022-09-27T16:31:47.7535600Z 
2022-09-27T16:31:47.7535999Z ----------------------------------------------------------------------
2022-09-27T16:31:47.7536341Z Ran 1 test in 4.460s
2022-09-27T16:31:47.7536525Z 
2022-09-27T16:31:47.7536636Z OK (skipped=1)
2022-09-27T16:31:47.7536794Z 
2022-09-27T16:31:47.7536905Z Generating XML reports...
2022-09-27T16:31:47.7593007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163143.xml
2022-09-27T16:31:48.0971717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:48.0972355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:48.0975180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:48.0975689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:48.1398805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:48.1399303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:48.1402101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:48.1402608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:48.1635834Z ok (6.039s)
2022-09-27T16:31:48.1636168Z 
2022-09-27T16:31:48.1636657Z ----------------------------------------------------------------------
2022-09-27T16:31:48.1636981Z Ran 1 test in 6.040s
2022-09-27T16:31:48.1637148Z 
2022-09-27T16:31:48.1637243Z OK
2022-09-27T16:31:48.1637378Z 
2022-09-27T16:31:48.1637507Z Generating XML reports...
2022-09-27T16:31:48.1673877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163142.xml
2022-09-27T16:31:48.6163665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:48.6563496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:48.6564553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:48.6569038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:48.6575621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:48.7577902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:49.9944031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:49.9944565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:49.9946879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:49.9947434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:50.0521451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qdd852k
2022-09-27T16:31:50.0522924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qdd852k/_remote_module_non_scriptable.py
2022-09-27T16:31:50.1497101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfs35z7p
2022-09-27T16:31:50.1498046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfs35z7p/_remote_module_non_scriptable.py
2022-09-27T16:31:50.1795300Z [1664296310.179174] [28fd6574525b:90611:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:50.1795928Z [1664296310.179174] [28fd6574525b:90611:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:50.1847870Z [1664296310.184507] [28fd6574525b:90612:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:50.1848590Z [1664296310.184507] [28fd6574525b:90612:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:50.2604738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:50.2605257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:50.2607742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:50.2608240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:50.5042654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:31:50.5066835Z 
2022-09-27T16:31:50.5067074Z Running tests...
2022-09-27T16:31:50.5067524Z ----------------------------------------------------------------------
2022-09-27T16:31:50.7644638Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:31:50.7668129Z 
2022-09-27T16:31:50.7668392Z Running tests...
2022-09-27T16:31:50.7668859Z ----------------------------------------------------------------------
2022-09-27T16:31:50.9874737Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:50.9875397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:50.9883681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:50.9884384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:51.0007446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:51.0008909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:51.0018854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:51.0019577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:31:51.6289569Z ok (6.886s)
2022-09-27T16:31:51.6289985Z 
2022-09-27T16:31:51.6290727Z ----------------------------------------------------------------------
2022-09-27T16:31:51.6291171Z Ran 1 test in 6.886s
2022-09-27T16:31:51.6291341Z 
2022-09-27T16:31:51.6291440Z OK
2022-09-27T16:31:51.6291567Z 
2022-09-27T16:31:51.6291686Z Generating XML reports...
2022-09-27T16:31:51.6327054Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163144.xml
2022-09-27T16:31:52.2420589Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:52.2621160Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90781
2022-09-27T16:31:52.2628004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90782
2022-09-27T16:31:52.4011827Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:52.4534900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90784
2022-09-27T16:31:52.4539498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90785
2022-09-27T16:31:53.8877477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:53.8878006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:53.8879734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:53.8880560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:53.9991694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:53.9992248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:53.9993933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:53.9994444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:54.0362296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:54.0362784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:54.0366125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:54.0366615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:54.2509783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:54.2510292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:54.2511751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:54.2512244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:54.2743675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:54.2744154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:54.2747935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:54.2748427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:54.3873566Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:31:54.3896185Z 
2022-09-27T16:31:54.3896465Z Running tests...
2022-09-27T16:31:54.3896904Z ----------------------------------------------------------------------
2022-09-27T16:31:54.5470542Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:54.5471233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:54.5472005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:54.5472707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:54.5578269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:54.6484621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:54.7657086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:54.7947466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:54.7962006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:54.7962971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:54.7969628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:54.8960612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:55.0706592Z skip: Skipped due to small world size. (4.563s)
2022-09-27T16:31:55.0706846Z 
2022-09-27T16:31:55.0707243Z ----------------------------------------------------------------------
2022-09-27T16:31:55.0707588Z Ran 1 test in 4.564s
2022-09-27T16:31:55.0707755Z 
2022-09-27T16:31:55.0707867Z OK (skipped=1)
2022-09-27T16:31:55.0708005Z 
2022-09-27T16:31:55.0708155Z Generating XML reports...
2022-09-27T16:31:55.0759193Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163150.xml
2022-09-27T16:31:55.3608432Z ok (4.594s)
2022-09-27T16:31:55.3608641Z 
2022-09-27T16:31:55.3609029Z ----------------------------------------------------------------------
2022-09-27T16:31:55.3609385Z Ran 1 test in 4.594s
2022-09-27T16:31:55.3609536Z 
2022-09-27T16:31:55.3609636Z OK
2022-09-27T16:31:55.3609776Z 
2022-09-27T16:31:55.3609913Z Generating XML reports...
2022-09-27T16:31:55.3646376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163150.xml
2022-09-27T16:31:56.0091200Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:56.0286902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90964
2022-09-27T16:31:56.0292876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90965
2022-09-27T16:31:57.4500746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:57.4501306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:57.4503717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:57.4504212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:57.6795894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:57.6798898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:57.6799512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:57.6799982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:57.7807240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:57.7807763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:57.7808867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:57.7809335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:57.8049544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:31:57.8050019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:31:57.8053343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:31:57.8053810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:31:57.9492464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:31:57.9515321Z 
2022-09-27T16:31:57.9515593Z Running tests...
2022-09-27T16:31:57.9516032Z ----------------------------------------------------------------------
2022-09-27T16:31:58.1814997Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:31:58.1838215Z 
2022-09-27T16:31:58.1838573Z Running tests...
2022-09-27T16:31:58.1839284Z ----------------------------------------------------------------------
2022-09-27T16:31:58.1858242Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:31:58.1858809Z 
2022-09-27T16:31:58.1859130Z ----------------------------------------------------------------------
2022-09-27T16:31:58.1859475Z Ran 1 test in 0.002s
2022-09-27T16:31:58.1859644Z 
2022-09-27T16:31:58.1859756Z OK (skipped=1)
2022-09-27T16:31:58.1859898Z 
2022-09-27T16:31:58.1860030Z Generating XML reports...
2022-09-27T16:31:58.1895699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163158.xml
2022-09-27T16:31:58.2858527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:31:58.3118092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:31:58.3118870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:58.3163695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:31:58.3170796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:31:58.4131248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:31:59.6796390Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:31:59.6869515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6oabngf7
2022-09-27T16:31:59.6872134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6oabngf7/_remote_module_non_scriptable.py
2022-09-27T16:31:59.7001055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91132
2022-09-27T16:31:59.7008114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91133
2022-09-27T16:31:59.7780909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcowoe3p
2022-09-27T16:31:59.7782080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcowoe3p/_remote_module_non_scriptable.py
2022-09-27T16:31:59.8036949Z [1664296319.803231] [28fd6574525b:90965:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:59.8037712Z [1664296319.803231] [28fd6574525b:90965:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:31:59.8038218Z [1664296319.803341] [28fd6574525b:90964:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:31:59.8038703Z [1664296319.803341] [28fd6574525b:90964:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:00.1938358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:00.1938881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:00.1940929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:00.1941422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:00.6126690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:00.6127722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:00.6137269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:00.6138916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:00.6259172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:00.6260202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:00.6268147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:00.6269512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:00.8149666Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:00.8173531Z 
2022-09-27T16:32:00.8173947Z Running tests...
2022-09-27T16:32:00.8175083Z ----------------------------------------------------------------------
2022-09-27T16:32:01.2414941Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... ok (6.851s)
2022-09-27T16:32:01.2415279Z 
2022-09-27T16:32:01.2415662Z ----------------------------------------------------------------------
2022-09-27T16:32:01.2416020Z Ran 1 test in 6.852s
2022-09-27T16:32:01.2416168Z 
2022-09-27T16:32:01.2416262Z OK
2022-09-27T16:32:01.2416399Z 
2022-09-27T16:32:01.2416527Z Generating XML reports...
2022-09-27T16:32:01.2451767Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163154.xml
2022-09-27T16:32:01.3530097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:01.3530573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:01.3531876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:01.3532377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:01.3975335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:01.3975829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:01.3977253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:01.3977736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:01.9414961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:01.9415511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:01.9416328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:01.9417265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:01.9423245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:01.9424471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:02.3073500Z skip: Skipped due to small world size. (4.355s)
2022-09-27T16:32:02.3073918Z 
2022-09-27T16:32:02.3074339Z ----------------------------------------------------------------------
2022-09-27T16:32:02.3074684Z Ran 1 test in 4.356s
2022-09-27T16:32:02.3074831Z 
2022-09-27T16:32:02.3074940Z OK (skipped=1)
2022-09-27T16:32:02.3075095Z 
2022-09-27T16:32:02.3075220Z Generating XML reports...
2022-09-27T16:32:02.3111575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163157.xml
2022-09-27T16:32:02.5958810Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:02.6130015Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.795s)
2022-09-27T16:32:02.6131242Z 
2022-09-27T16:32:02.6131544Z ----------------------------------------------------------------------
2022-09-27T16:32:02.6131878Z Ran 1 test in 1.796s
2022-09-27T16:32:02.6132044Z 
2022-09-27T16:32:02.6132155Z OK (skipped=1)
2022-09-27T16:32:02.6132315Z 
2022-09-27T16:32:02.6132426Z Generating XML reports...
2022-09-27T16:32:02.6165386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163200.xml
2022-09-27T16:32:03.4156296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:03.4160825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:03.4162784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:03.4164393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:04.0054560Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:04.0077339Z 
2022-09-27T16:32:04.0077925Z Running tests...
2022-09-27T16:32:04.0078432Z ----------------------------------------------------------------------
2022-09-27T16:32:04.0110183Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T16:32:04.0110564Z 
2022-09-27T16:32:04.0110825Z ----------------------------------------------------------------------
2022-09-27T16:32:04.0111158Z Ran 1 test in 0.003s
2022-09-27T16:32:04.0111325Z 
2022-09-27T16:32:04.0111435Z OK (skipped=1)
2022-09-27T16:32:04.0111593Z 
2022-09-27T16:32:04.0111736Z Generating XML reports...
2022-09-27T16:32:04.0145729Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163204.xml
2022-09-27T16:32:04.4928806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:04.4929313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:04.4930578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:04.4931054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:04.8037118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:04.8037626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:04.8039607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:04.8040129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:05.0423143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:05.0445671Z 
2022-09-27T16:32:05.0445930Z Running tests...
2022-09-27T16:32:05.0446363Z ----------------------------------------------------------------------
2022-09-27T16:32:05.3596514Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:05.3619361Z 
2022-09-27T16:32:05.3619851Z Running tests...
2022-09-27T16:32:05.3620316Z ----------------------------------------------------------------------
2022-09-27T16:32:05.3637974Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:32:05.3638306Z 
2022-09-27T16:32:05.3638844Z ----------------------------------------------------------------------
2022-09-27T16:32:05.3639191Z Ran 1 test in 0.002s
2022-09-27T16:32:05.3639339Z 
2022-09-27T16:32:05.3639452Z OK (skipped=1)
2022-09-27T16:32:05.3639959Z 
2022-09-27T16:32:05.3640086Z Generating XML reports...
2022-09-27T16:32:05.3674991Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163205.xml
2022-09-27T16:32:05.8844155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:05.8845148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:05.8846689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:05.8847605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:06.3802451Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:06.3826208Z 
2022-09-27T16:32:06.3826759Z Running tests...
2022-09-27T16:32:06.3827819Z ----------------------------------------------------------------------
2022-09-27T16:32:06.6602496Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:06.6802541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91388
2022-09-27T16:32:06.6810093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91389
2022-09-27T16:32:07.4080879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:07.4084701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:07.4087424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:07.4089783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:07.9419002Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:07.9440692Z 
2022-09-27T16:32:07.9441023Z Running tests...
2022-09-27T16:32:07.9441470Z ----------------------------------------------------------------------
2022-09-27T16:32:07.9458716Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:32:07.9459825Z 
2022-09-27T16:32:07.9460525Z ----------------------------------------------------------------------
2022-09-27T16:32:07.9461007Z Ran 1 test in 0.002s
2022-09-27T16:32:07.9461172Z 
2022-09-27T16:32:07.9461283Z OK (skipped=1)
2022-09-27T16:32:07.9461438Z 
2022-09-27T16:32:07.9461547Z Generating XML reports...
2022-09-27T16:32:07.9491843Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:07.9494639Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163207.xml
2022-09-27T16:32:07.9703471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91454
2022-09-27T16:32:07.9708597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91455
2022-09-27T16:32:08.3473741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:08.3474254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:08.3475931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:08.3476406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:08.3898098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:08.3898584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:08.3900566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:08.3901039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:09.0144332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:09.0148953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:09.0153132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:09.0245580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:09.0335905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:09.1162120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:09.1370852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:32:09.1371414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:32:09.1372232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:32:09.1372928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:32:09.6436120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:09.6436616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:09.6439264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:09.6439759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:09.6562521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:09.6562994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:09.6566833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:09.6567328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:09.9117119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:09.9117615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:09.9119672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:09.9120360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:10.1509863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:10.1584400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:10.1585235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:10.1610937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:10.1617730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:10.2598453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:10.4173553Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:10.4197142Z 
2022-09-27T16:32:10.4197665Z Running tests...
2022-09-27T16:32:10.4198228Z ----------------------------------------------------------------------
2022-09-27T16:32:10.4216664Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-09-27T16:32:10.4217144Z 
2022-09-27T16:32:10.4217779Z ----------------------------------------------------------------------
2022-09-27T16:32:10.4218116Z Ran 1 test in 0.002s
2022-09-27T16:32:10.4218292Z 
2022-09-27T16:32:10.4218402Z OK (skipped=1)
2022-09-27T16:32:10.4218542Z 
2022-09-27T16:32:10.4218668Z Generating XML reports...
2022-09-27T16:32:10.4253968Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163210.xml
2022-09-27T16:32:10.6917744Z ok (5.647s)
2022-09-27T16:32:10.6917956Z 
2022-09-27T16:32:10.6918350Z ----------------------------------------------------------------------
2022-09-27T16:32:10.6918694Z Ran 1 test in 5.647s
2022-09-27T16:32:10.6918870Z 
2022-09-27T16:32:10.6918949Z OK
2022-09-27T16:32:10.6919087Z 
2022-09-27T16:32:10.6919230Z Generating XML reports...
2022-09-27T16:32:10.6975067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163205.xml
2022-09-27T16:32:11.5371557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yfovj26
2022-09-27T16:32:11.5372705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yfovj26/_remote_module_non_scriptable.py
2022-09-27T16:32:11.6303109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bz05m7i
2022-09-27T16:32:11.6305125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bz05m7i/_remote_module_non_scriptable.py
2022-09-27T16:32:11.6645541Z [1664296331.664101] [28fd6574525b:91454:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:11.6646693Z [1664296331.664101] [28fd6574525b:91454:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:11.6665182Z [1664296331.666214] [28fd6574525b:91455:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:11.6665670Z [1664296331.666214] [28fd6574525b:91455:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:12.4831739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:12.4832241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:12.4835309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:12.4835807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:12.5811695Z ok (6.198s)
2022-09-27T16:32:12.5811918Z 
2022-09-27T16:32:12.5812291Z ----------------------------------------------------------------------
2022-09-27T16:32:12.5812652Z Ran 1 test in 6.199s
2022-09-27T16:32:12.5812818Z 
2022-09-27T16:32:12.5813191Z OK
2022-09-27T16:32:12.5813348Z 
2022-09-27T16:32:12.5813485Z Generating XML reports...
2022-09-27T16:32:12.5847467Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163206.xml
2022-09-27T16:32:12.9868150Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:12.9892133Z 
2022-09-27T16:32:12.9892460Z Running tests...
2022-09-27T16:32:12.9892917Z ----------------------------------------------------------------------
2022-09-27T16:32:13.0034325Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:13.0034874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:13.0038298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:13.0038808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:13.5114882Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:13.5138485Z 
2022-09-27T16:32:13.5138878Z Running tests...
2022-09-27T16:32:13.5139559Z ----------------------------------------------------------------------
2022-09-27T16:32:14.6124433Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:14.6125022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:14.6127889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:14.6128399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:14.6374873Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:14.6548475Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.665s)
2022-09-27T16:32:14.6549057Z 
2022-09-27T16:32:14.6549355Z ----------------------------------------------------------------------
2022-09-27T16:32:14.6549674Z Ran 1 test in 1.665s
2022-09-27T16:32:14.6549841Z 
2022-09-27T16:32:14.6549949Z OK (skipped=1)
2022-09-27T16:32:14.6550103Z 
2022-09-27T16:32:14.6550227Z Generating XML reports...
2022-09-27T16:32:14.6584323Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163212.xml
2022-09-27T16:32:15.0932975Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:15.1127229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91684
2022-09-27T16:32:15.1133295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91685
2022-09-27T16:32:15.1732520Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:15.1756434Z 
2022-09-27T16:32:15.1756732Z Running tests...
2022-09-27T16:32:15.1757196Z ----------------------------------------------------------------------
2022-09-27T16:32:16.7937264Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:16.7937915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:16.7938516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:16.7939009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:16.8008400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:16.8008883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:16.8012065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:16.8012569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:16.8250369Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:16.8443134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91784
2022-09-27T16:32:16.8450262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91785
2022-09-27T16:32:16.9005999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:16.9006499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:16.9008642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:16.9009132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:17.3245931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:17.3246738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:17.3247532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:17.3248241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:17.3353824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:17.4075104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:17.4106812Z 
2022-09-27T16:32:17.4107276Z Running tests...
2022-09-27T16:32:17.4108058Z ----------------------------------------------------------------------
2022-09-27T16:32:17.4257705Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:17.4669062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:17.4669616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:17.4670361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:17.4671075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:18.4747921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:18.4748999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:18.4750213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:18.4751205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:18.5162466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:18.5163558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:18.5164794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:18.5165789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:18.9220533Z ok (5.408s)
2022-09-27T16:32:18.9220933Z 
2022-09-27T16:32:18.9221598Z ----------------------------------------------------------------------
2022-09-27T16:32:18.9222595Z Ran 1 test in 5.408s
2022-09-27T16:32:18.9222937Z 
2022-09-27T16:32:18.9223088Z OK
2022-09-27T16:32:18.9223352Z 
2022-09-27T16:32:18.9223596Z Generating XML reports...
2022-09-27T16:32:18.9259218Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163213.xml
2022-09-27T16:32:19.0306006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:19.0424352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:19.0425914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:19.0457950Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:19.0508574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:19.0515298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:19.0632896Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.652s)
2022-09-27T16:32:19.0634430Z 
2022-09-27T16:32:19.0634995Z ----------------------------------------------------------------------
2022-09-27T16:32:19.0635639Z Ran 1 test in 1.652s
2022-09-27T16:32:19.0635953Z 
2022-09-27T16:32:19.0636152Z OK (skipped=1)
2022-09-27T16:32:19.0636454Z 
2022-09-27T16:32:19.0636663Z Generating XML reports...
2022-09-27T16:32:19.0670804Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163217.xml
2022-09-27T16:32:19.1439929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:20.4690569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum3idwm2
2022-09-27T16:32:20.4691177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum3idwm2/_remote_module_non_scriptable.py
2022-09-27T16:32:20.5210011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69x2sbe7
2022-09-27T16:32:20.5210608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69x2sbe7/_remote_module_non_scriptable.py
2022-09-27T16:32:20.5570413Z [1664296340.556546] [28fd6574525b:91784:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:20.5570920Z [1664296340.556546] [28fd6574525b:91784:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:20.5613571Z [1664296340.561038] [28fd6574525b:91785:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:20.5614075Z [1664296340.561038] [28fd6574525b:91785:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:21.0970860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:21.0971443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:21.0972458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:21.0972949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:21.2398470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:21.2399247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:21.2400971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:21.2401732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:21.3231804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:21.3232360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:21.6182042Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:21.6206070Z 
2022-09-27T16:32:21.6206532Z Running tests...
2022-09-27T16:32:21.6207052Z ----------------------------------------------------------------------
2022-09-27T16:32:21.7524326Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:21.7546646Z 
2022-09-27T16:32:21.7546926Z Running tests...
2022-09-27T16:32:21.7547358Z ----------------------------------------------------------------------
2022-09-27T16:32:21.8574071Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... ok (6.681s)
2022-09-27T16:32:21.8574432Z 
2022-09-27T16:32:21.8574782Z ----------------------------------------------------------------------
2022-09-27T16:32:21.8575380Z Ran 1 test in 6.682s
2022-09-27T16:32:21.8575532Z 
2022-09-27T16:32:21.8575631Z OK
2022-09-27T16:32:21.8575768Z 
2022-09-27T16:32:21.8575895Z Generating XML reports...
2022-09-27T16:32:21.8610390Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163215.xml
2022-09-27T16:32:23.3996679Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:23.4195469Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91984
2022-09-27T16:32:23.4202080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91985
2022-09-27T16:32:23.4248823Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:23.4411990Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.686s)
2022-09-27T16:32:23.4412575Z 
2022-09-27T16:32:23.4412852Z ----------------------------------------------------------------------
2022-09-27T16:32:23.4413184Z Ran 1 test in 1.687s
2022-09-27T16:32:23.4413351Z 
2022-09-27T16:32:23.4413445Z OK (skipped=1)
2022-09-27T16:32:23.4413603Z 
2022-09-27T16:32:23.4413731Z Generating XML reports...
2022-09-27T16:32:23.4446944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163221.xml
2022-09-27T16:32:24.0624239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:24.0627401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:24.0630687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:24.0632166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:24.6833622Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:24.6857440Z 
2022-09-27T16:32:24.6857849Z Running tests...
2022-09-27T16:32:24.6858358Z ----------------------------------------------------------------------
2022-09-27T16:32:24.6906046Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.005s)
2022-09-27T16:32:24.6906700Z 
2022-09-27T16:32:24.6907066Z ----------------------------------------------------------------------
2022-09-27T16:32:24.6907400Z Ran 1 test in 0.005s
2022-09-27T16:32:24.6907548Z 
2022-09-27T16:32:24.6907659Z OK (skipped=1)
2022-09-27T16:32:24.6907814Z 
2022-09-27T16:32:24.6907940Z Generating XML reports...
2022-09-27T16:32:24.6943180Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163224.xml
2022-09-27T16:32:25.0946559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:25.0947073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:25.0978673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:25.0979186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:25.1310411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:25.1311015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:25.1313720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:25.1314247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:25.5509886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:25.5511296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:25.5513042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:25.5514009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:25.6537192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:25.6538266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:25.6539719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:25.6541158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:25.6547845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:25.6551180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:26.0266521Z skip: Skipped due to small world size. (4.406s)
2022-09-27T16:32:26.0266762Z 
2022-09-27T16:32:26.0267351Z ----------------------------------------------------------------------
2022-09-27T16:32:26.0267706Z Ran 1 test in 4.406s
2022-09-27T16:32:26.0267876Z 
2022-09-27T16:32:26.0267991Z OK (skipped=1)
2022-09-27T16:32:26.0268154Z 
2022-09-27T16:32:26.0268296Z Generating XML reports...
2022-09-27T16:32:26.0305563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163221.xml
2022-09-27T16:32:26.0498025Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:26.0520632Z 
2022-09-27T16:32:26.0521003Z Running tests...
2022-09-27T16:32:26.0521787Z ----------------------------------------------------------------------
2022-09-27T16:32:26.5517066Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:26.5517910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:26.5520096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:26.5520830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:27.0434833Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:27.0457635Z 
2022-09-27T16:32:27.0458162Z Running tests...
2022-09-27T16:32:27.0458916Z ----------------------------------------------------------------------
2022-09-27T16:32:27.0495913Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.004s)
2022-09-27T16:32:27.0496541Z 
2022-09-27T16:32:27.0496963Z ----------------------------------------------------------------------
2022-09-27T16:32:27.0497347Z Ran 1 test in 0.004s
2022-09-27T16:32:27.0497519Z 
2022-09-27T16:32:27.0497643Z OK (skipped=1)
2022-09-27T16:32:27.0497800Z 
2022-09-27T16:32:27.0497930Z Generating XML reports...
2022-09-27T16:32:27.0532579Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163227.xml
2022-09-27T16:32:27.6720567Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:27.6913447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92160
2022-09-27T16:32:27.6919940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92161
2022-09-27T16:32:28.1908850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:28.1909354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:28.1912230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:28.1912731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:28.7619045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:28.7642386Z 
2022-09-27T16:32:28.7642857Z Running tests...
2022-09-27T16:32:28.7643352Z ----------------------------------------------------------------------
2022-09-27T16:32:29.0260939Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:29.0261553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:29.0262730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:29.0263211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:29.3894586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:29.3895104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:29.3897469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:29.3897974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:29.4353023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:29.4353518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:29.4356175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:29.4356652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:29.5253954Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:29.5277408Z 
2022-09-27T16:32:29.5277556Z Running tests...
2022-09-27T16:32:29.5278622Z ----------------------------------------------------------------------
2022-09-27T16:32:29.9067184Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:29.9702529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:29.9703405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:29.9777765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:29.9784382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:30.0717256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:30.4180504Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:30.4393865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92270
2022-09-27T16:32:30.4398844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92271
2022-09-27T16:32:31.1975995Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:31.2179612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92335
2022-09-27T16:32:31.2185954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92336
2022-09-27T16:32:31.3482063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjb9el78z
2022-09-27T16:32:31.3483112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjb9el78z/_remote_module_non_scriptable.py
2022-09-27T16:32:31.4551293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp586wc_zy
2022-09-27T16:32:31.4552324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp586wc_zy/_remote_module_non_scriptable.py
2022-09-27T16:32:32.0065921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:32.0079434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:32.1806499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:32.1807110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:32.1809962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:32.1810461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:32.1901472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:32.1902134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:32.1903847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:32.1904331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:32.6027467Z ok (6.550s)
2022-09-27T16:32:32.6027728Z 
2022-09-27T16:32:32.6028139Z ----------------------------------------------------------------------
2022-09-27T16:32:32.6028486Z Ran 1 test in 6.551s
2022-09-27T16:32:32.6028656Z 
2022-09-27T16:32:32.6028753Z OK
2022-09-27T16:32:32.6028873Z 
2022-09-27T16:32:32.6029015Z Generating XML reports...
2022-09-27T16:32:32.6071823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163226.xml
2022-09-27T16:32:32.7299573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:32.7300164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:32.7300960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:32.7301665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:32.7307400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:32.7307902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:32.8551338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:32.8552252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:32.8552909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:32.8553404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:32.8640705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:32.8641213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:32.8644354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:32.8644844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:33.1467575Z ok (4.382s)
2022-09-27T16:32:33.1467787Z 
2022-09-27T16:32:33.1468181Z ----------------------------------------------------------------------
2022-09-27T16:32:33.1468521Z Ran 1 test in 4.382s
2022-09-27T16:32:33.1468691Z 
2022-09-27T16:32:33.1468787Z OK
2022-09-27T16:32:33.1468906Z 
2022-09-27T16:32:33.1469039Z Generating XML reports...
2022-09-27T16:32:33.1504106Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163228.xml
2022-09-27T16:32:33.3883632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:33.3959445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:33.3961311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:33.3985193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:33.3991542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:33.4093591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphq9irj3v
2022-09-27T16:32:33.4095352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphq9irj3v/_remote_module_non_scriptable.py
2022-09-27T16:32:33.4974874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:33.5077376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyuns175f
2022-09-27T16:32:33.5079569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyuns175f/_remote_module_non_scriptable.py
2022-09-27T16:32:34.7809064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:34.7809614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:34.7811390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:34.7811888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:34.9948921Z [1664296354.994374] [28fd6574525b:92335:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:34.9949520Z [1664296354.994374] [28fd6574525b:92335:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:35.0056041Z [1664296355.005265] [28fd6574525b:92336:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:35.0056534Z [1664296355.005265] [28fd6574525b:92336:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:35.3010127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:35.3027027Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3028876Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3031084Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T16:32:35.3032541Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:32:35.3034109Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T16:32:35.3035295Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:32:35.3035777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:35.3036064Z 
2022-09-27T16:32:35.3036305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:35.3036673Z Running tests...
2022-09-27T16:32:35.3037084Z ----------------------------------------------------------------------
2022-09-27T16:32:35.3038231Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3039754Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3043399Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3044925Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3049238Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3050761Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3054966Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3056465Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3060025Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3061515Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:32:35.3528920Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:35.3530088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:35.3531245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:35.3532197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:35.8612686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:35.8634581Z 
2022-09-27T16:32:35.8634912Z Running tests...
2022-09-27T16:32:35.8635383Z ----------------------------------------------------------------------
2022-09-27T16:32:35.9292962Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... ok (6.401s)
2022-09-27T16:32:35.9293325Z 
2022-09-27T16:32:35.9293688Z ----------------------------------------------------------------------
2022-09-27T16:32:35.9294024Z Ran 1 test in 6.401s
2022-09-27T16:32:35.9294191Z 
2022-09-27T16:32:35.9294288Z OK
2022-09-27T16:32:35.9294406Z 
2022-09-27T16:32:35.9294535Z Generating XML reports...
2022-09-27T16:32:35.9329716Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163229.xml
2022-09-27T16:32:37.1152510Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:37.1444548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92536
2022-09-27T16:32:37.1450005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92537
2022-09-27T16:32:37.4787734Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:37.4980079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92539
2022-09-27T16:32:37.4986187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92540
2022-09-27T16:32:38.1297108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:38.1297620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:38.1300643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:38.1301123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:38.6790786Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:38.6813396Z 
2022-09-27T16:32:38.6813738Z Running tests...
2022-09-27T16:32:38.6814192Z ----------------------------------------------------------------------
2022-09-27T16:32:38.6859104Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.004s)
2022-09-27T16:32:38.6859680Z 
2022-09-27T16:32:38.6860247Z ----------------------------------------------------------------------
2022-09-27T16:32:38.6860590Z Ran 1 test in 0.005s
2022-09-27T16:32:38.6860766Z 
2022-09-27T16:32:38.6860881Z OK (skipped=1)
2022-09-27T16:32:38.6861038Z 
2022-09-27T16:32:38.6861166Z Generating XML reports...
2022-09-27T16:32:38.6895613Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163238.xml
2022-09-27T16:32:38.8184188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:38.8184698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:38.8186005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:38.8186500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:38.8477295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:38.8477808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:38.8478395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:38.8478861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:39.1940854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:39.1941852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:39.1943054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:39.1944025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:39.2443844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:39.2444767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:39.2448408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:39.2449385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:39.3315661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:39.3786441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:39.3788022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:39.3822449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:39.3828985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:39.4800663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:39.8276308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:39.8277403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:39.8278887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:39.8280616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:39.8385119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:39.9290722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:40.4057175Z ok (4.542s)
2022-09-27T16:32:40.4057554Z 
2022-09-27T16:32:40.4058368Z ----------------------------------------------------------------------
2022-09-27T16:32:40.4058717Z Ran 1 test in 4.542s
2022-09-27T16:32:40.4058888Z 
2022-09-27T16:32:40.4058964Z OK
2022-09-27T16:32:40.4059101Z 
2022-09-27T16:32:40.4059239Z Generating XML reports...
2022-09-27T16:32:40.4093910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163235.xml
2022-09-27T16:32:40.6789222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:40.6789733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:40.6791246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:40.6791745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:40.7543493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggl7kxso
2022-09-27T16:32:40.7544588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggl7kxso/_remote_module_non_scriptable.py
2022-09-27T16:32:40.8542962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6kuo9es
2022-09-27T16:32:40.8544372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6kuo9es/_remote_module_non_scriptable.py
2022-09-27T16:32:41.1910189Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:41.1933387Z 
2022-09-27T16:32:41.1933544Z Running tests...
2022-09-27T16:32:41.1934489Z ----------------------------------------------------------------------
2022-09-27T16:32:41.1965503Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T16:32:41.1966263Z 
2022-09-27T16:32:41.1966817Z ----------------------------------------------------------------------
2022-09-27T16:32:41.1967179Z Ran 1 test in 0.003s
2022-09-27T16:32:41.1967340Z 
2022-09-27T16:32:41.1967451Z OK (skipped=1)
2022-09-27T16:32:41.1967608Z 
2022-09-27T16:32:41.1967736Z Generating XML reports...
2022-09-27T16:32:41.2001748Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163241.xml
2022-09-27T16:32:41.3950701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:41.3951592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:41.3957485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:41.3960152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:41.9558477Z ok (6.652s)
2022-09-27T16:32:41.9558743Z 
2022-09-27T16:32:41.9559107Z ----------------------------------------------------------------------
2022-09-27T16:32:41.9559446Z Ran 1 test in 6.652s
2022-09-27T16:32:41.9559627Z 
2022-09-27T16:32:41.9559721Z OK
2022-09-27T16:32:41.9559856Z 
2022-09-27T16:32:41.9559973Z Generating XML reports...
2022-09-27T16:32:41.9594986Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163235.xml
2022-09-27T16:32:42.5938630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:42.5939127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:42.5940952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:42.5941445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:43.1347422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:43.1369346Z 
2022-09-27T16:32:43.1369757Z Running tests...
2022-09-27T16:32:43.1370705Z ----------------------------------------------------------------------
2022-09-27T16:32:43.1393905Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:32:43.1394263Z 
2022-09-27T16:32:43.1394590Z ----------------------------------------------------------------------
2022-09-27T16:32:43.1394911Z Ran 1 test in 0.002s
2022-09-27T16:32:43.1395080Z 
2022-09-27T16:32:43.1395192Z OK (skipped=1)
2022-09-27T16:32:43.1395349Z 
2022-09-27T16:32:43.1395476Z Generating XML reports...
2022-09-27T16:32:43.1428688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163243.xml
2022-09-27T16:32:43.2306800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:43.2307339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:43.2309561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:43.2310066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:43.7395351Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:43.7419357Z 
2022-09-27T16:32:43.7419688Z Running tests...
2022-09-27T16:32:43.7420165Z ----------------------------------------------------------------------
2022-09-27T16:32:43.9949879Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:43.9950479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:43.9951646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:43.9952146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:44.4906138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:44.4928921Z 
2022-09-27T16:32:44.4929366Z Running tests...
2022-09-27T16:32:44.4930227Z ----------------------------------------------------------------------
2022-09-27T16:32:44.9927087Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:44.9928261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:44.9929456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:44.9930869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:45.3668334Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:45.3873584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92865
2022-09-27T16:32:45.3878967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92866
2022-09-27T16:32:45.4916437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:45.4940533Z 
2022-09-27T16:32:45.4941083Z Running tests...
2022-09-27T16:32:45.4941616Z ----------------------------------------------------------------------
2022-09-27T16:32:45.4969964Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T16:32:45.4970592Z 
2022-09-27T16:32:45.4971228Z ----------------------------------------------------------------------
2022-09-27T16:32:45.4971876Z Ran 1 test in 0.003s
2022-09-27T16:32:45.4972201Z 
2022-09-27T16:32:45.4972398Z OK (skipped=1)
2022-09-27T16:32:45.4972718Z 
2022-09-27T16:32:45.4972965Z Generating XML reports...
2022-09-27T16:32:45.5009048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163245.xml
2022-09-27T16:32:46.0452591Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:46.0817283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92931
2022-09-27T16:32:46.0904669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92932
2022-09-27T16:32:47.0777682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:47.0778604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:47.0779793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:47.0780273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:47.1100113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:47.1100600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:47.1104072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:47.1104558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:47.5655250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:47.5656094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:47.5658492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:47.5658975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:47.5856217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:47.6318829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:47.6319626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:47.6365394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:47.6372069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:47.7333908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:47.7383282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:47.7383762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:47.7388035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:47.7388566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:47.7723739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:47.7724506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:47.7727472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:47.7727973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:48.0727637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:48.0751124Z 
2022-09-27T16:32:48.0751443Z Running tests...
2022-09-27T16:32:48.0751903Z ----------------------------------------------------------------------
2022-09-27T16:32:48.0778492Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T16:32:48.0778825Z 
2022-09-27T16:32:48.0779114Z ----------------------------------------------------------------------
2022-09-27T16:32:48.0779737Z Ran 1 test in 0.003s
2022-09-27T16:32:48.0779907Z 
2022-09-27T16:32:48.0780026Z OK (skipped=1)
2022-09-27T16:32:48.0780186Z 
2022-09-27T16:32:48.0780313Z Generating XML reports...
2022-09-27T16:32:48.0814701Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163248.xml
2022-09-27T16:32:48.2930491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:48.3041290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:48.3042143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:48.3135533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:48.3141722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:48.4057259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:48.9943114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7edfz7eh
2022-09-27T16:32:48.9944308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7edfz7eh/_remote_module_non_scriptable.py
2022-09-27T16:32:49.1664484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6ewox4u
2022-09-27T16:32:49.1668843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6ewox4u/_remote_module_non_scriptable.py
2022-09-27T16:32:49.1968916Z [1664296369.196399] [28fd6574525b:92865:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:49.1969422Z [1664296369.196399] [28fd6574525b:92865:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:49.2073411Z [1664296369.206983] [28fd6574525b:92866:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:49.2073917Z [1664296369.206983] [28fd6574525b:92866:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:49.6604335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j463ix_
2022-09-27T16:32:49.6605938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j463ix_/_remote_module_non_scriptable.py
2022-09-27T16:32:49.7357089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxckntzs
2022-09-27T16:32:49.7357978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxckntzs/_remote_module_non_scriptable.py
2022-09-27T16:32:50.0180069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.0180639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.0371387Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpv6ewox4u'>
2022-09-27T16:32:50.0371894Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:32:50.0372506Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp7edfz7eh'>
2022-09-27T16:32:50.0372952Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:32:50.0939506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:50.0940022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:50.0941131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:50.0941615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:50.2911957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.2913446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.2921313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.2922689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.3045707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.3046764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.3053503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.3054512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:32:50.5994050Z ok (6.857s)
2022-09-27T16:32:50.5994455Z 
2022-09-27T16:32:50.5995127Z ----------------------------------------------------------------------
2022-09-27T16:32:50.5995774Z Ran 1 test in 6.857s
2022-09-27T16:32:50.5996078Z 
2022-09-27T16:32:50.5996248Z OK
2022-09-27T16:32:50.5996521Z 
2022-09-27T16:32:50.5996741Z Generating XML reports...
2022-09-27T16:32:50.6031331Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163243.xml
2022-09-27T16:32:50.6405554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:50.6431146Z 
2022-09-27T16:32:50.6431583Z Running tests...
2022-09-27T16:32:50.6432080Z ----------------------------------------------------------------------
2022-09-27T16:32:50.6454382Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:32:50.6454720Z 
2022-09-27T16:32:50.6455007Z ----------------------------------------------------------------------
2022-09-27T16:32:50.6455482Z Ran 1 test in 0.002s
2022-09-27T16:32:50.6455782Z 
2022-09-27T16:32:50.6455878Z OK (skipped=1)
2022-09-27T16:32:50.6456036Z 
2022-09-27T16:32:50.6456164Z Generating XML reports...
2022-09-27T16:32:50.6491471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163250.xml
2022-09-27T16:32:50.9018270Z ok (6.409s)
2022-09-27T16:32:50.9018514Z 
2022-09-27T16:32:50.9019128Z ----------------------------------------------------------------------
2022-09-27T16:32:50.9019502Z Ran 1 test in 6.409s
2022-09-27T16:32:50.9019653Z 
2022-09-27T16:32:50.9019756Z OK
2022-09-27T16:32:50.9019894Z 
2022-09-27T16:32:50.9020027Z Generating XML reports...
2022-09-27T16:32:50.9073028Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163244.xml
2022-09-27T16:32:52.6650293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:52.6650854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:52.6651463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:52.6651957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:52.7679299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:52.7679814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:52.7682719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:52.7683220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:53.0917046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:53.0917553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:53.0919571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:53.0920079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:53.1425164Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:53.1447946Z 
2022-09-27T16:32:53.1448430Z Running tests...
2022-09-27T16:32:53.1448947Z ----------------------------------------------------------------------
2022-09-27T16:32:53.1467362Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:32:53.1467690Z 
2022-09-27T16:32:53.1467985Z ----------------------------------------------------------------------
2022-09-27T16:32:53.1468321Z Ran 1 test in 0.002s
2022-09-27T16:32:53.1468497Z 
2022-09-27T16:32:53.1468609Z OK (skipped=1)
2022-09-27T16:32:53.1468770Z 
2022-09-27T16:32:53.1468898Z Generating XML reports...
2022-09-27T16:32:53.1501994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163253.xml
2022-09-27T16:32:53.2508481Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:32:53.2531596Z 
2022-09-27T16:32:53.2532023Z Running tests...
2022-09-27T16:32:53.2532538Z ----------------------------------------------------------------------
2022-09-27T16:32:53.6039228Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:32:53.6062580Z 
2022-09-27T16:32:53.6063022Z Running tests...
2022-09-27T16:32:53.6063528Z ----------------------------------------------------------------------
2022-09-27T16:32:54.8610484Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:54.8811377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93234
2022-09-27T16:32:54.8817343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93235
2022-09-27T16:32:55.0738344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:55.0738888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:55.0740653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:55.0741247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:55.2444527Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:32:55.2644938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93238
2022-09-27T16:32:55.2652315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93239
2022-09-27T16:32:55.6115752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:55.6153818Z 
2022-09-27T16:32:55.6154622Z Running tests...
2022-09-27T16:32:55.6156022Z ----------------------------------------------------------------------
2022-09-27T16:32:55.6195854Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.004s)
2022-09-27T16:32:55.6198071Z 
2022-09-27T16:32:55.6202568Z ----------------------------------------------------------------------
2022-09-27T16:32:55.6204967Z Ran 1 test in 0.005s
2022-09-27T16:32:55.6207768Z 
2022-09-27T16:32:55.6211035Z OK (skipped=1)
2022-09-27T16:32:55.6213944Z 
2022-09-27T16:32:55.6214771Z Generating XML reports...
2022-09-27T16:32:55.6270638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163255.xml
2022-09-27T16:32:56.5892721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:56.5893564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:56.5894701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:56.5895177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:56.5958658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:56.5959121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:56.5962230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:56.5962701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:56.9608165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:56.9608962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:56.9609608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:56.9610339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:56.9999291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:56.9999778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:57.0003177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:57.0003658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:57.1162179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:57.1501889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:57.1502784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:57.1567326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:57.1574289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:57.2515881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:57.4727551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:32:57.5046955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:32:57.5048233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:57.5133283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:32:57.5140290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:32:57.5634747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:32:57.5635393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:32:57.5636918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:32:57.5637442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:32:57.6060237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:32:58.0964497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:32:58.0987993Z 
2022-09-27T16:32:58.0988279Z Running tests...
2022-09-27T16:32:58.0989225Z ----------------------------------------------------------------------
2022-09-27T16:32:58.1014531Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-09-27T16:32:58.1014846Z 
2022-09-27T16:32:58.1015147Z ----------------------------------------------------------------------
2022-09-27T16:32:58.1015477Z Ran 1 test in 0.003s
2022-09-27T16:32:58.1015643Z 
2022-09-27T16:32:58.1015765Z OK (skipped=1)
2022-09-27T16:32:58.1015899Z 
2022-09-27T16:32:58.1016034Z Generating XML reports...
2022-09-27T16:32:58.1051700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163258.xml
2022-09-27T16:32:58.5220056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvtvq3cr
2022-09-27T16:32:58.5220688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvtvq3cr/_remote_module_non_scriptable.py
2022-09-27T16:32:58.6252702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kebazni
2022-09-27T16:32:58.6254183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kebazni/_remote_module_non_scriptable.py
2022-09-27T16:32:58.6536361Z [1664296378.653093] [28fd6574525b:93234:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:58.6537374Z [1664296378.653093] [28fd6574525b:93234:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:58.6598363Z [1664296378.659442] [28fd6574525b:93235:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:32:58.6599344Z [1664296378.659442] [28fd6574525b:93235:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:32:58.8583645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lgwurox
2022-09-27T16:32:58.8584818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lgwurox/_remote_module_non_scriptable.py
2022-09-27T16:32:58.9133908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dizh2w2
2022-09-27T16:32:58.9134975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dizh2w2/_remote_module_non_scriptable.py
2022-09-27T16:33:00.1260122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:00.1260647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:00.1261624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:00.1262101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:00.6247527Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:33:00.6268926Z 
2022-09-27T16:33:00.6269561Z Running tests...
2022-09-27T16:33:00.6270072Z ----------------------------------------------------------------------
2022-09-27T16:33:00.6290111Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-09-27T16:33:00.6290444Z 
2022-09-27T16:33:00.6290733Z ----------------------------------------------------------------------
2022-09-27T16:33:00.6291053Z Ran 1 test in 0.002s
2022-09-27T16:33:00.6291222Z 
2022-09-27T16:33:00.6291334Z OK (skipped=1)
2022-09-27T16:33:00.6291495Z 
2022-09-27T16:33:00.6291623Z Generating XML reports...
2022-09-27T16:33:00.6325153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163300.xml
2022-09-27T16:33:02.5056010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:33:02.5056684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:33:02.5058068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:33:02.5058878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:33:02.9994194Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:33:03.0016264Z 
2022-09-27T16:33:03.0016749Z Running tests...
2022-09-27T16:33:03.0017291Z ----------------------------------------------------------------------
2022-09-27T16:33:59.5013874Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... [1664296439.500875] [28fd6574525b:93235:1]    ucc_schedule.h:182  UCC  WARN  timeout 60 sec has expired on req 0x555e25f78840, seq_num 6, TL_NCCL, team_id 1, size 2, rank 1, ctx_rank 1: Allreduce Cuda inplace=1 bytes=40 float32 sum
2022-09-27T16:33:59.5023291Z [1664296439.502000] [28fd6574525b:93235:1]    ucc_schedule.h:182  UCC  WARN  timeout 60 sec has expired on req 0x555e25f78600, seq_num 7, TL_NCCL, team_id 1, size 2, rank 1, ctx_rank 1: Allreduce Cuda inplace=1 bytes=40 float32 sum
2022-09-27T16:34:07.4733789Z [E ProcessGroupNCCL.cpp:821] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=8, OpType=ALLREDUCE, Timeout(ms)=60000) ran for 67987 milliseconds before timing out.
2022-09-27T16:34:08.3460146Z [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
2022-09-27T16:34:08.3581948Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2022-09-27T16:34:08.3582332Z Traceback (most recent call last):
2022-09-27T16:34:08.3583001Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:34:08.3583438Z     getattr(self, test_name)()
2022-09-27T16:34:08.3587388Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:34:08.3587796Z     fn()
2022-09-27T16:34:08.3588330Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 145, in wrapper
2022-09-27T16:34:08.3588736Z     return func(*args, **kwargs)
2022-09-27T16:34:08.3589340Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 8700, in test_ddp_buffer_hook_allreduce_return_future
2022-09-27T16:34:08.3589825Z     self._test_ddp_buffer_hook_allreduce(
2022-09-27T16:34:08.3590436Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 8681, in _test_ddp_buffer_hook_allreduce
2022-09-27T16:34:08.3592867Z     loss_hook.backward()
2022-09-27T16:34:08.3594014Z   File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 482, in backward
2022-09-27T16:34:08.3595069Z     torch.autograd.backward(
2022-09-27T16:34:08.3595620Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
2022-09-27T16:34:08.3596093Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:34:08.3596771Z RuntimeError: NCCL communicator was aborted on rank 0.  Original reason for failure was: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=8, OpType=ALLREDUCE, Timeout(ms)=60000) ran for 67987 milliseconds before timing out.
2022-09-27T16:34:08.3597542Z Exception raised from getNcclComm at /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/NCCLUtils.cpp:23 (most recent call first):
2022-09-27T16:34:08.3598427Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f639827850b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-09-27T16:34:08.3599412Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f6398273ede in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-09-27T16:34:08.3600666Z frame #2: <unknown function> + 0x1ab0ca (0x7f63adc210ca in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-09-27T16:34:08.3601342Z frame #3: <unknown function> + 0x1bb5d4 (0x7f63adc315d4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-09-27T16:34:08.3602228Z frame #4: c10d::ProcessGroupNCCL::allreduce_impl(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllreduceOptions const&) + 0x30 (0x7f63adc32540 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-09-27T16:34:08.3603225Z frame #5: c10d::ProcessGroupNCCL::allreduce(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllreduceOptions const&) + 0x2c9 (0x7f63adc36dd9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-09-27T16:34:08.3604007Z frame #6: <unknown function> + 0x46e89bd (0x7f63a61189bd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3604636Z frame #7: <unknown function> + 0x46ee784 (0x7f63a611e784 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3605645Z frame #8: c10d::ops::allreduce(c10::intrusive_ptr<c10d::ProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroup> > const&, c10::ArrayRef<at::Tensor>, c10d::AllreduceOptions const&) + 0x2cf (0x7f63a611be9f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3606500Z frame #9: <unknown function> + 0x47496a5 (0x7f63a61796a5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3607199Z frame #10: c10d::Reducer::run_comm_hook(c10d::GradBucket&) + 0x8c (0x7f63a6187a9c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3607950Z frame #11: c10d::Reducer::all_reduce_bucket(c10d::Reducer::Bucket&) + 0xa7 (0x7f63a6189d17 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3608680Z frame #12: c10d::Reducer::mark_bucket_ready(unsigned long) + 0x6b (0x7f63a6189e4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3609414Z frame #13: c10d::Reducer::mark_variable_ready(unsigned long) + 0x193 (0x7f63a6192ad3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3610140Z frame #14: c10d::Reducer::autograd_hook(unsigned long) + 0x1ad (0x7f63a6192e0d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3610853Z frame #15: <unknown function> + 0x4763121 (0x7f63a6193121 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3611588Z frame #16: <unknown function> + 0x476532a (0x7f63a619532a in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3612605Z frame #17: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x6da (0x7f63a5765dfa in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3613596Z frame #18: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x60e (0x7f63a57677ee in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3614473Z frame #19: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x99 (0x7f63a575e659 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.3615419Z frame #20: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x6c (0x7f63afcf533c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-09-27T16:34:08.3616014Z frame #21: <unknown function> + 0xc9039 (0x7f63c788a039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-09-27T16:34:08.3616658Z frame #22: <unknown function> + 0x76db (0x7f63e7e3a6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-09-27T16:34:08.3617156Z frame #23: clone + 0x3f (0x7f63e7b6361f in /lib/x86_64-linux-gnu/libc.so.6)
2022-09-27T16:34:08.3617383Z 
2022-09-27T16:34:08.3617524Z  exiting process 0 with exit code: 10
2022-09-27T16:34:08.3751470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:08.3752265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:08.3763377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:08.8032138Z Process 0 terminated with exit code 10, terminating remaining processes.
2022-09-27T16:34:08.8064171Z ERROR (75.200s)
2022-09-27T16:34:08.8064788Z     test_ddp_buffer_hook_allreduce_return_future errored - num_retries_left: 3
2022-09-27T16:34:08.8065163Z Traceback (most recent call last):
2022-09-27T16:34:08.8065708Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 501, in wrapper
2022-09-27T16:34:08.8066144Z     self._join_processes(fn)
2022-09-27T16:34:08.8066690Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 724, in _join_processes
2022-09-27T16:34:08.8067129Z     self._check_return_codes(elapsed_time)
2022-09-27T16:34:08.8067680Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 769, in _check_return_codes
2022-09-27T16:34:08.8068094Z     raise RuntimeError(error)
2022-09-27T16:34:08.8068447Z RuntimeError: Process 0 exited with error code 10 and exception:
2022-09-27T16:34:08.8068776Z Traceback (most recent call last):
2022-09-27T16:34:08.8069314Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-09-27T16:34:08.8069722Z     getattr(self, test_name)()
2022-09-27T16:34:08.8070227Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-09-27T16:34:08.8070618Z     fn()
2022-09-27T16:34:08.8071110Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 145, in wrapper
2022-09-27T16:34:08.8071502Z     return func(*args, **kwargs)
2022-09-27T16:34:08.8072104Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 8700, in test_ddp_buffer_hook_allreduce_return_future
2022-09-27T16:34:08.8072582Z     self._test_ddp_buffer_hook_allreduce(
2022-09-27T16:34:08.8073193Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 8681, in _test_ddp_buffer_hook_allreduce
2022-09-27T16:34:08.8073863Z     loss_hook.backward()
2022-09-27T16:34:08.8074351Z   File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 482, in backward
2022-09-27T16:34:08.8074729Z     torch.autograd.backward(
2022-09-27T16:34:08.8075219Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
2022-09-27T16:34:08.8075699Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:34:08.8076357Z RuntimeError: NCCL communicator was aborted on rank 0.  Original reason for failure was: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=8, OpType=ALLREDUCE, Timeout(ms)=60000) ran for 67987 milliseconds before timing out.
2022-09-27T16:34:08.8077084Z Exception raised from getNcclComm at /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/NCCLUtils.cpp:23 (most recent call first):
2022-09-27T16:34:08.8077974Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f639827850b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-09-27T16:34:08.8078958Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f6398273ede in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-09-27T16:34:08.8079865Z frame #2: <unknown function> + 0x1ab0ca (0x7f63adc210ca in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-09-27T16:34:08.8081187Z frame #3: <unknown function> + 0x1bb5d4 (0x7f63adc315d4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-09-27T16:34:08.8082076Z frame #4: c10d::ProcessGroupNCCL::allreduce_impl(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllreduceOptions const&) + 0x30 (0x7f63adc32540 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-09-27T16:34:08.8083075Z frame #5: c10d::ProcessGroupNCCL::allreduce(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllreduceOptions const&) + 0x2c9 (0x7f63adc36dd9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-09-27T16:34:08.8083852Z frame #6: <unknown function> + 0x46e89bd (0x7f63a61189bd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8084475Z frame #7: <unknown function> + 0x46ee784 (0x7f63a611e784 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8085465Z frame #8: c10d::ops::allreduce(c10::intrusive_ptr<c10d::ProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroup> > const&, c10::ArrayRef<at::Tensor>, c10d::AllreduceOptions const&) + 0x2cf (0x7f63a611be9f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8086307Z frame #9: <unknown function> + 0x47496a5 (0x7f63a61796a5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8087010Z frame #10: c10d::Reducer::run_comm_hook(c10d::GradBucket&) + 0x8c (0x7f63a6187a9c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8087761Z frame #11: c10d::Reducer::all_reduce_bucket(c10d::Reducer::Bucket&) + 0xa7 (0x7f63a6189d17 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8088489Z frame #12: c10d::Reducer::mark_bucket_ready(unsigned long) + 0x6b (0x7f63a6189e4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8089221Z frame #13: c10d::Reducer::mark_variable_ready(unsigned long) + 0x193 (0x7f63a6192ad3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8089944Z frame #14: c10d::Reducer::autograd_hook(unsigned long) + 0x1ad (0x7f63a6192e0d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8090714Z frame #15: <unknown function> + 0x4763121 (0x7f63a6193121 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8091350Z frame #16: <unknown function> + 0x476532a (0x7f63a619532a in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8092344Z frame #17: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x6da (0x7f63a5765dfa in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8093335Z frame #18: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x60e (0x7f63a57677ee in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8094217Z frame #19: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x99 (0x7f63a575e659 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-09-27T16:34:08.8095163Z frame #20: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x6c (0x7f63afcf533c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-09-27T16:34:08.8095835Z frame #21: <unknown function> + 0xc9039 (0x7f63c788a039 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-09-27T16:34:08.8096385Z frame #22: <unknown function> + 0x76db (0x7f63e7e3a6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-09-27T16:34:08.8096890Z frame #23: clone + 0x3f (0x7f63e7b6361f in /lib/x86_64-linux-gnu/libc.so.6)
2022-09-27T16:34:08.8097118Z 
2022-09-27T16:34:08.8097138Z 
2022-09-27T16:34:08.8097156Z 
2022-09-27T16:34:08.8097175Z 
2022-09-27T16:34:08.8098546Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93504
2022-09-27T16:34:08.8107407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93505
2022-09-27T16:34:11.9165112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:11.9165628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:11.9166691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:11.9167185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:11.9697884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:11.9698369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:11.9701815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:11.9702313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:12.8793504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:13.4273793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:13.4392396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:13.4393244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:13.4477851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:13.4485486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:13.5407706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:13.7100633Z ok (80.456s)
2022-09-27T16:34:13.7100829Z 
2022-09-27T16:34:13.7101223Z ----------------------------------------------------------------------
2022-09-27T16:34:13.7101761Z Ran 1 test in 80.457s
2022-09-27T16:34:13.7101951Z 
2022-09-27T16:34:13.7102045Z OK
2022-09-27T16:34:13.7102179Z 
2022-09-27T16:34:13.7102316Z Generating XML reports...
2022-09-27T16:34:13.7138059Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163253.xml
2022-09-27T16:34:14.4871387Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:14.5076272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93580
2022-09-27T16:34:14.5081098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93581
2022-09-27T16:34:14.8501570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2w1e8gcx
2022-09-27T16:34:14.8502227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2w1e8gcx/_remote_module_non_scriptable.py
2022-09-27T16:34:14.9636113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1mxhn7u
2022-09-27T16:34:14.9637542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1mxhn7u/_remote_module_non_scriptable.py
2022-09-27T16:34:15.5950001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:15.5950936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:15.5958901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:15.5959635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:15.6068716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:15.6069220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:15.6077655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:15.6078162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:16.0268204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:16.0268736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:16.0270646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:16.0271131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:16.2251347Z ok (7.418s)
2022-09-27T16:34:16.2251718Z 
2022-09-27T16:34:16.2252350Z ----------------------------------------------------------------------
2022-09-27T16:34:16.2252969Z Ran 2 tests in 82.619s
2022-09-27T16:34:16.2253268Z 
2022-09-27T16:34:16.2253526Z OK (skipped=1, expected failures=1)
2022-09-27T16:34:16.2253874Z 
2022-09-27T16:34:16.2254114Z Generating XML reports...
2022-09-27T16:34:16.2295098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163253.xml
2022-09-27T16:34:16.2729690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:16.2730677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:16.2732144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:16.2733057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:16.3269088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:16.3270011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:16.3271096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:16.3271916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:16.5168329Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:16.5189421Z 
2022-09-27T16:34:16.5189893Z Running tests...
2022-09-27T16:34:16.5190413Z ----------------------------------------------------------------------
2022-09-27T16:34:16.8662862Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:16.8663525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:16.8664317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:16.8665020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:16.8771361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:16.8838305Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9677275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:16.9742427Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9755169Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9755748Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9770822Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9779105Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9791603Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9792177Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9803831Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9810939Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9822032Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9822622Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9834012Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9836937Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9848255Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9848845Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9860084Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9864087Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9874645Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9875241Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9887056Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9889772Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9900470Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9901091Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9912366Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9916330Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9927069Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9927660Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9939025Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9941522Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9952361Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9952950Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9964797Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9968425Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9979225Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9979810Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:16.9991271Z STAGE:2022-09-27 16:34:16 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:16.9993969Z STAGE:2022-09-27 16:34:16 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:17.0004698Z STAGE:2022-09-27 16:34:17 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:17.0005291Z STAGE:2022-09-27 16:34:17 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:17.0016803Z STAGE:2022-09-27 16:34:17 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:17.0020556Z STAGE:2022-09-27 16:34:17 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:17.0031350Z STAGE:2022-09-27 16:34:17 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:17.0043866Z STAGE:2022-09-27 16:34:17 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:17.0044444Z STAGE:2022-09-27 16:34:17 93580:93580 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:17.0046834Z STAGE:2022-09-27 16:34:17 93581:93581 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:17.0057432Z STAGE:2022-09-27 16:34:17 93581:93581 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:17.0058022Z STAGE:2022-09-27 16:34:17 93580:93580 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:17.5154266Z ok (74.513s)
2022-09-27T16:34:17.5154499Z 
2022-09-27T16:34:17.5154890Z ----------------------------------------------------------------------
2022-09-27T16:34:17.5155238Z Ran 1 test in 74.514s
2022-09-27T16:34:17.5155391Z 
2022-09-27T16:34:17.5155485Z OK
2022-09-27T16:34:17.5155626Z 
2022-09-27T16:34:17.5155767Z Generating XML reports...
2022-09-27T16:34:17.5190232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163302.xml
2022-09-27T16:34:18.1728863Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:18.1920842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93736
2022-09-27T16:34:18.1927125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93737
2022-09-27T16:34:18.3552338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:18.3553006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:18.3554856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:18.3555349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:18.9149047Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:34:18.9191269Z 
2022-09-27T16:34:18.9192860Z Running tests...
2022-09-27T16:34:18.9194644Z ----------------------------------------------------------------------
2022-09-27T16:34:19.6610924Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:19.6612212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:19.6614014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:19.6615386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:19.9151896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:19.9152844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:19.9154047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:19.9154993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:19.9195422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:19.9196370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:19.9198555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:19.9199530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:20.1598492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:34:20.1623083Z 
2022-09-27T16:34:20.1623509Z Running tests...
2022-09-27T16:34:20.1624021Z ----------------------------------------------------------------------
2022-09-27T16:34:20.4505773Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:20.4506418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:20.4507212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:20.4507924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:20.4514518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:20.4515013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:20.4518094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:34:20.4518611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:34:20.4519277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:20.4519974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:20.5732455Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:20.5938014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93842
2022-09-27T16:34:20.5943543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93843
2022-09-27T16:34:21.8632871Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:21.8737950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1p19g4a
2022-09-27T16:34:21.8739225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1p19g4a/_remote_module_non_scriptable.py
2022-09-27T16:34:21.8801340Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.718s)
2022-09-27T16:34:21.8801936Z 
2022-09-27T16:34:21.8802244Z ----------------------------------------------------------------------
2022-09-27T16:34:21.8802586Z Ran 1 test in 1.718s
2022-09-27T16:34:21.8802753Z 
2022-09-27T16:34:21.8802864Z OK (skipped=1)
2022-09-27T16:34:21.8803005Z 
2022-09-27T16:34:21.8803383Z Generating XML reports...
2022-09-27T16:34:21.8837618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163420.xml
2022-09-27T16:34:21.9230308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzz6mf41c
2022-09-27T16:34:21.9231656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzz6mf41c/_remote_module_non_scriptable.py
2022-09-27T16:34:21.9523676Z [1664296461.951777] [28fd6574525b:93737:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:34:21.9524760Z [1664296461.951791] [28fd6574525b:93736:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:34:21.9525719Z [1664296461.951777] [28fd6574525b:93737:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:34:21.9526649Z [1664296461.951791] [28fd6574525b:93736:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:34:22.2584841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:22.2585794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:22.2587019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:22.2587936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:22.3396758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:22.3397649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:22.3398707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:22.3399573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:22.7788238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:22.8055870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:22.8056386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:22.8068005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:22.8068498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:22.8684606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:22.8685616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:22.8700228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:22.8707990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:22.9708263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:23.7051787Z ok (7.186s)
2022-09-27T16:34:23.7052094Z 
2022-09-27T16:34:23.7052825Z ----------------------------------------------------------------------
2022-09-27T16:34:23.7053495Z Ran 1 test in 7.186s
2022-09-27T16:34:23.7053711Z 
2022-09-27T16:34:23.7053807Z OK
2022-09-27T16:34:23.7053950Z 
2022-09-27T16:34:23.7054085Z Generating XML reports...
2022-09-27T16:34:23.7089130Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163416.xml
2022-09-27T16:34:24.0081810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:24.0082350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:24.0084426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:24.0085280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:24.3388530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zr2albz
2022-09-27T16:34:24.3389476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zr2albz/_remote_module_non_scriptable.py
2022-09-27T16:34:24.4040051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeb1egz6z
2022-09-27T16:34:24.4041503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeb1egz6z/_remote_module_non_scriptable.py
2022-09-27T16:34:24.5532140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:34:24.5554194Z 
2022-09-27T16:34:24.5554646Z Running tests...
2022-09-27T16:34:24.5555590Z ----------------------------------------------------------------------
2022-09-27T16:34:25.5241429Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... <class 'torch.nn.parameter.Parameter'>
2022-09-27T16:34:25.9403133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:25.9403679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:25.9406014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:25.9406985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:26.1066999Z ok (7.187s)
2022-09-27T16:34:26.1067206Z 
2022-09-27T16:34:26.1067569Z ----------------------------------------------------------------------
2022-09-27T16:34:26.1067930Z Ran 1 test in 7.188s
2022-09-27T16:34:26.1068101Z 
2022-09-27T16:34:26.1068184Z OK
2022-09-27T16:34:26.1068325Z 
2022-09-27T16:34:26.1068459Z Generating XML reports...
2022-09-27T16:34:26.1104012Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163418.xml
2022-09-27T16:34:26.2022658Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:26.2221451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94005
2022-09-27T16:34:26.2227633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94006
2022-09-27T16:34:26.4843171Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:26.4867274Z 
2022-09-27T16:34:26.4867766Z Running tests...
2022-09-27T16:34:26.4868252Z ----------------------------------------------------------------------
2022-09-27T16:34:27.9066264Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:27.9067525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:27.9068325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:27.9068797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:27.9148439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:27.9149199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:27.9152586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:27.9153346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:28.1969067Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:28.2166824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94105
2022-09-27T16:34:28.2172722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94106
2022-09-27T16:34:28.2774914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:28.2775585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:28.2777166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:28.2778002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:28.4404872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:28.4405828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:28.4406633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:28.4407343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:28.4513310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:28.5417974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:28.5527375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:34:28.5528519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:34:28.5529328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:28.5530171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:34:28.5625311Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5625826Z 
2022-09-27T16:34:28.5638675Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5639157Z 
2022-09-27T16:34:28.5653357Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5660657Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5672340Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5673603Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5684862Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5691606Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5702367Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5703333Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5714692Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5717692Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5729825Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5730320Z 
2022-09-27T16:34:28.5741232Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5744280Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5754761Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5755736Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5767488Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5770646Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5781671Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5782654Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5793616Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5796916Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5807871Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5808848Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5820171Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5823295Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5834329Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5835314Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5847182Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5849718Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5860267Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5861249Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5872480Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5875752Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5887199Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5887654Z 
2022-09-27T16:34:28.5898663Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5901529Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5912374Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5913403Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5924887Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5928208Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:34:28.5939171Z STAGE:2022-09-27 16:34:28 94006:94006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.5940156Z STAGE:2022-09-27 16:34:28 94005:94005 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:34:28.7723832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:34:28.7747741Z 
2022-09-27T16:34:28.7748286Z Running tests...
2022-09-27T16:34:28.7749239Z ----------------------------------------------------------------------
2022-09-27T16:34:29.0294267Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... ok (4.474s)
2022-09-27T16:34:29.0294996Z 
2022-09-27T16:34:29.0295702Z ----------------------------------------------------------------------
2022-09-27T16:34:29.0296050Z Ran 1 test in 4.474s
2022-09-27T16:34:29.0296219Z 
2022-09-27T16:34:29.0296320Z OK
2022-09-27T16:34:29.0296460Z 
2022-09-27T16:34:29.0296585Z Generating XML reports...
2022-09-27T16:34:29.0332268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163424.xml
2022-09-27T16:34:29.8917071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:29.8917573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:29.8918909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:29.8919394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:29.8939502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:29.8939954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:29.8944615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:29.8945105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:30.4082330Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:30.4279372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94224
2022-09-27T16:34:30.4286178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94225
2022-09-27T16:34:30.4287492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:30.4834425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:30.4835901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:30.4898313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:30.4905276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:30.4907403Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:34:30.5849392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:30.5850595Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:34:31.1339189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:31.1342585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:31.1344672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:31.1346621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:31.6670016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:34:31.6693315Z 
2022-09-27T16:34:31.6693617Z Running tests...
2022-09-27T16:34:31.6694063Z ----------------------------------------------------------------------
2022-09-27T16:34:31.9008354Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15m5r6x3
2022-09-27T16:34:31.9009821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15m5r6x3/_remote_module_non_scriptable.py
2022-09-27T16:34:31.9761777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf6yxv1q
2022-09-27T16:34:31.9763497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf6yxv1q/_remote_module_non_scriptable.py
2022-09-27T16:34:32.0052459Z [1664296472.004836] [28fd6574525b:94105:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:34:32.0052972Z [1664296472.004836] [28fd6574525b:94105:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:34:32.0085376Z [1664296472.008211] [28fd6574525b:94106:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:34:32.0085852Z [1664296472.008211] [28fd6574525b:94106:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:34:32.0820881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:32.0821382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:32.0823048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:32.0823542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:32.0945877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:32.0946341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:32.0949895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:32.0950380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:32.6139229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:32.6214559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:32.6215371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:32.6240518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:32.6247900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:32.7231686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:32.8665962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:32.8666978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:32.8677025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:32.8678102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:32.8906108Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:34:32.8906740Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:34:33.1090971Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:34:33.1091571Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:34:33.1162506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:33.1163022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:33.1175695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:33.1176186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:33.1387418Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:34:33.1388159Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:34:33.2808992Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:34:33.2813008Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:34:33.2878163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:33.2878673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:33.2891230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:33.2891733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:33.3010822Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:33.3212378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94318
2022-09-27T16:34:33.3217325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94319
2022-09-27T16:34:33.9583312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwwst90o
2022-09-27T16:34:33.9584491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwwst90o/_remote_module_non_scriptable.py
2022-09-27T16:34:34.1311531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa746j2w2
2022-09-27T16:34:34.1312678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa746j2w2/_remote_module_non_scriptable.py
2022-09-27T16:34:34.2321589Z ok (7.745s)
2022-09-27T16:34:34.2321983Z 
2022-09-27T16:34:34.2322657Z ----------------------------------------------------------------------
2022-09-27T16:34:34.2323307Z Ran 1 test in 7.745s
2022-09-27T16:34:34.2323621Z 
2022-09-27T16:34:34.2323789Z OK
2022-09-27T16:34:34.2324050Z 
2022-09-27T16:34:34.2324297Z Generating XML reports...
2022-09-27T16:34:34.2358273Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163426.xml
2022-09-27T16:34:35.0223684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:35.0224683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:35.0226220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:35.0227211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:35.0445577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:35.0446534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:35.0448035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:35.0448990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:35.6463613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:35.6464653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:35.6466098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:35.6467447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:35.6472457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:35.6473385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:35.9425499Z ok (7.167s)
2022-09-27T16:34:35.9425877Z 
2022-09-27T16:34:35.9426509Z ----------------------------------------------------------------------
2022-09-27T16:34:35.9427134Z Ran 1 test in 7.168s
2022-09-27T16:34:35.9427438Z 
2022-09-27T16:34:35.9427605Z OK
2022-09-27T16:34:35.9427835Z 
2022-09-27T16:34:35.9428075Z Generating XML reports...
2022-09-27T16:34:35.9462019Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163428.xml
2022-09-27T16:34:36.0283793Z skip: Skipped due to small world size. (4.359s)
2022-09-27T16:34:36.0284256Z 
2022-09-27T16:34:36.0284895Z ----------------------------------------------------------------------
2022-09-27T16:34:36.0285416Z Ran 1 test in 4.359s
2022-09-27T16:34:36.0285585Z 
2022-09-27T16:34:36.0285695Z OK (skipped=1)
2022-09-27T16:34:36.0285854Z 
2022-09-27T16:34:36.0285988Z Generating XML reports...
2022-09-27T16:34:36.0320867Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163431.xml
2022-09-27T16:34:36.3730797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:36.3731768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:36.3732910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:36.3733767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:36.8613239Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:36.8635076Z 
2022-09-27T16:34:36.8635841Z Running tests...
2022-09-27T16:34:36.8636364Z ----------------------------------------------------------------------
2022-09-27T16:34:38.0944824Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:38.0945421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:38.0946831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:38.0947319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:38.1628457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:38.1628958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:38.1631154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:38.1631662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:38.4771529Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:38.4966678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94501
2022-09-27T16:34:38.4972280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94502
2022-09-27T16:34:38.6206517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:34:38.6230970Z 
2022-09-27T16:34:38.6231469Z Running tests...
2022-09-27T16:34:38.6231946Z ----------------------------------------------------------------------
2022-09-27T16:34:38.6555424Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:34:38.6578990Z 
2022-09-27T16:34:38.6579286Z Running tests...
2022-09-27T16:34:38.6580049Z ----------------------------------------------------------------------
2022-09-27T16:34:40.1532094Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:40.1532697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:40.1534001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:40.1534495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:40.2046224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:40.2046706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:40.2049670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:40.2050210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:40.3320299Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:40.3332882Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:40.3515069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94570
2022-09-27T16:34:40.3521696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94572
2022-09-27T16:34:40.3527670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94573
2022-09-27T16:34:40.3534633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94574
2022-09-27T16:34:40.6649302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:40.7157606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:40.7158412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:40.7257277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:40.7264541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:40.7267446Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:34:40.8173049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:40.8174383Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:34:42.0560714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:42.0561310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:42.0562415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:42.0562894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:42.0563475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:42.0563977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:42.0564585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:42.0565374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:42.0696667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:42.0697184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:42.0699842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:42.0700335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:42.1196443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:42.1196970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:42.1198723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:42.1199212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:42.1212645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3pu06py
2022-09-27T16:34:42.1215632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3pu06py/_remote_module_non_scriptable.py
2022-09-27T16:34:42.2756112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphoe5r63q
2022-09-27T16:34:42.2757063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphoe5r63q/_remote_module_non_scriptable.py
2022-09-27T16:34:42.3097082Z [1664296482.309235] [28fd6574525b:94501:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:34:42.3097618Z [1664296482.309235] [28fd6574525b:94501:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:34:42.3114481Z [1664296482.311145] [28fd6574525b:94502:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:34:42.3114990Z [1664296482.311145] [28fd6574525b:94502:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:34:42.6097358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:42.6156020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:42.6158731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:42.6159546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:42.6260345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:42.6266613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:42.6626044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:42.6626864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:42.6706926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:42.6713997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:42.7175662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:42.7640862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:43.1634580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:43.1635147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:43.1646519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:43.1647026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:43.1654036Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:34:43.1654618Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:34:43.1681399Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:34:43.1682019Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:34:43.1684267Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:34:43.1684981Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:34:43.1685994Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:34:43.1686681Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:34:43.4689725Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:34:43.4692893Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:34:43.4761584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:43.4762117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:43.4772538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:43.4773045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:43.4778982Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:34:43.4779868Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:34:43.4806529Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:34:43.4807367Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:34:43.4808216Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:34:43.4808902Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:34:44.1434920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskl6jt9a
2022-09-27T16:34:44.1435831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskl6jt9a/_remote_module_non_scriptable.py
2022-09-27T16:34:44.1545282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvvvc79o
2022-09-27T16:34:44.1547125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvvvc79o/_remote_module_non_scriptable.py
2022-09-27T16:34:44.4104176Z ok (7.547s)
2022-09-27T16:34:44.4104393Z 
2022-09-27T16:34:44.4105112Z ----------------------------------------------------------------------
2022-09-27T16:34:44.4105791Z Ran 1 test in 7.547s
2022-09-27T16:34:44.4106023Z 
2022-09-27T16:34:44.4106125Z OK
2022-09-27T16:34:44.4106262Z 
2022-09-27T16:34:44.4106398Z Generating XML reports...
2022-09-27T16:34:44.4149528Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163436.xml
2022-09-27T16:34:44.6638055Z ok (6.005s)
2022-09-27T16:34:44.6638433Z 
2022-09-27T16:34:44.6639207Z ----------------------------------------------------------------------
2022-09-27T16:34:44.6639741Z Ran 1 test in 6.006s
2022-09-27T16:34:44.6639913Z 
2022-09-27T16:34:44.6639990Z OK
2022-09-27T16:34:44.6640129Z 
2022-09-27T16:34:44.6640552Z Generating XML reports...
2022-09-27T16:34:44.6675768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163438.xml
2022-09-27T16:34:46.0049000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:46.0049544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:34:46.5656774Z ok (7.942s)
2022-09-27T16:34:46.5656970Z 
2022-09-27T16:34:46.5657385Z ----------------------------------------------------------------------
2022-09-27T16:34:46.5657732Z Ran 1 test in 7.943s
2022-09-27T16:34:46.5657902Z 
2022-09-27T16:34:46.5658000Z OK
2022-09-27T16:34:46.5658134Z 
2022-09-27T16:34:46.5658254Z Generating XML reports...
2022-09-27T16:34:46.5694723Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163438.xml
2022-09-27T16:34:46.6466582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:46.6467101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:46.6469131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:46.6469636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:46.7969614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:46.7970124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:46.7971696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:46.7972442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:47.1526614Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:47.1550306Z 
2022-09-27T16:34:47.1550648Z Running tests...
2022-09-27T16:34:47.1551377Z ----------------------------------------------------------------------
2022-09-27T16:34:47.1571675Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: flaky on PyTorch CI: No such file or directory: '/tmp/checkpoint.pt' (0.002s)
2022-09-27T16:34:47.1572275Z 
2022-09-27T16:34:47.1572991Z ----------------------------------------------------------------------
2022-09-27T16:34:47.1573720Z Ran 1 test in 0.002s
2022-09-27T16:34:47.1574057Z 
2022-09-27T16:34:47.1574251Z OK (skipped=1)
2022-09-27T16:34:47.1574566Z 
2022-09-27T16:34:47.1574813Z Generating XML reports...
2022-09-27T16:34:47.1609839Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163447.xml
2022-09-27T16:34:47.2879216Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:34:47.2901046Z 
2022-09-27T16:34:47.2901310Z Running tests...
2022-09-27T16:34:47.2901763Z ----------------------------------------------------------------------
2022-09-27T16:34:48.6644339Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:48.6644932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:48.6657799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:48.6658307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:48.8493242Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:48.8655529Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.575s)
2022-09-27T16:34:48.8656117Z 
2022-09-27T16:34:48.8656392Z ----------------------------------------------------------------------
2022-09-27T16:34:48.8656714Z Ran 1 test in 1.575s
2022-09-27T16:34:48.8656884Z 
2022-09-27T16:34:48.8656995Z OK (skipped=1)
2022-09-27T16:34:48.8657152Z 
2022-09-27T16:34:48.8657280Z Generating XML reports...
2022-09-27T16:34:48.8690469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163447.xml
2022-09-27T16:34:49.1558281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:49.1558827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:49.1561129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:49.1561806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:49.1674765Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:34:49.1700216Z 
2022-09-27T16:34:49.1700554Z Running tests...
2022-09-27T16:34:49.1700991Z ----------------------------------------------------------------------
2022-09-27T16:34:49.6597810Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:49.6620138Z 
2022-09-27T16:34:49.6620292Z Running tests...
2022-09-27T16:34:49.6620962Z ----------------------------------------------------------------------
2022-09-27T16:34:49.6641894Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:34:49.6642355Z 
2022-09-27T16:34:49.6642642Z ----------------------------------------------------------------------
2022-09-27T16:34:49.6642965Z Ran 1 test in 0.002s
2022-09-27T16:34:49.6643135Z 
2022-09-27T16:34:49.6643247Z OK (skipped=1)
2022-09-27T16:34:49.6643404Z 
2022-09-27T16:34:49.6643533Z Generating XML reports...
2022-09-27T16:34:49.6677227Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163449.xml
2022-09-27T16:34:50.8945001Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:50.9142628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94949
2022-09-27T16:34:50.9149696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94950
2022-09-27T16:34:50.9800675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:50.9801210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:50.9802644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:50.9803474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:51.4703237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:34:51.4727640Z 
2022-09-27T16:34:51.4728140Z Running tests...
2022-09-27T16:34:51.4728619Z ----------------------------------------------------------------------
2022-09-27T16:34:51.6537032Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:51.6537711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:51.6538568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:51.6539070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:52.1487063Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:52.1508954Z 
2022-09-27T16:34:52.1509286Z Running tests...
2022-09-27T16:34:52.1509727Z ----------------------------------------------------------------------
2022-09-27T16:34:52.1529641Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:34:52.1530084Z 
2022-09-27T16:34:52.1530350Z ----------------------------------------------------------------------
2022-09-27T16:34:52.1530689Z Ran 1 test in 0.002s
2022-09-27T16:34:52.1530858Z 
2022-09-27T16:34:52.1530969Z OK (skipped=1)
2022-09-27T16:34:52.1531128Z 
2022-09-27T16:34:52.1531259Z Generating XML reports...
2022-09-27T16:34:52.1565155Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163452.xml
2022-09-27T16:34:52.5689588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:52.5690162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:52.5691227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:52.5691905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:52.5743243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:52.5743906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:52.5747585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:52.5748120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:53.0872015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:34:53.0878743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:34:53.0885905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:53.0974284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:34:53.1063950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:34:53.1157955Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:53.1336940Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.660s)
2022-09-27T16:34:53.1338996Z 
2022-09-27T16:34:53.1339653Z ----------------------------------------------------------------------
2022-09-27T16:34:53.1340441Z Ran 1 test in 1.660s
2022-09-27T16:34:53.1340831Z 
2022-09-27T16:34:53.1341093Z OK (skipped=1)
2022-09-27T16:34:53.1341462Z 
2022-09-27T16:34:53.1341721Z Generating XML reports...
2022-09-27T16:34:53.1373838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163451.xml
2022-09-27T16:34:53.1970939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:34:54.1581640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:54.1582141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:54.1583741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:54.1584233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:54.5061216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3urmhwj7
2022-09-27T16:34:54.5061844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3urmhwj7/_remote_module_non_scriptable.py
2022-09-27T16:34:54.5111827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5av3rbz
2022-09-27T16:34:54.5114759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5av3rbz/_remote_module_non_scriptable.py
2022-09-27T16:34:54.7234904Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:54.7258402Z 
2022-09-27T16:34:54.7258690Z Running tests...
2022-09-27T16:34:54.7259150Z ----------------------------------------------------------------------
2022-09-27T16:34:54.7280457Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:34:54.7281296Z 
2022-09-27T16:34:54.7281585Z ----------------------------------------------------------------------
2022-09-27T16:34:54.7281908Z Ran 1 test in 0.002s
2022-09-27T16:34:54.7282076Z 
2022-09-27T16:34:54.7282192Z OK (skipped=1)
2022-09-27T16:34:54.7282350Z 
2022-09-27T16:34:54.7282478Z Generating XML reports...
2022-09-27T16:34:54.7317463Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163454.xml
2022-09-27T16:34:55.0154187Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:34:55.2748024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:55.2748540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:55.2749853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:55.2750325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:55.6257376Z ok (6.455s)
2022-09-27T16:34:55.6257600Z 
2022-09-27T16:34:55.6257990Z ----------------------------------------------------------------------
2022-09-27T16:34:55.6258349Z Ran 1 test in 6.456s
2022-09-27T16:34:55.6258500Z 
2022-09-27T16:34:55.6258599Z OK
2022-09-27T16:34:55.6258746Z 
2022-09-27T16:34:55.6258887Z Generating XML reports...
2022-09-27T16:34:55.6294562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163449.xml
2022-09-27T16:34:55.7689746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:34:55.7713698Z 
2022-09-27T16:34:55.7714573Z Running tests...
2022-09-27T16:34:55.7715604Z ----------------------------------------------------------------------
2022-09-27T16:34:56.7349127Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:56.7350670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:56.7352150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:56.7353309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:57.2271358Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:57.2295574Z 
2022-09-27T16:34:57.2295930Z Running tests...
2022-09-27T16:34:57.2296428Z ----------------------------------------------------------------------
2022-09-27T16:34:57.2317310Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:34:57.2317829Z 
2022-09-27T16:34:57.2318123Z ----------------------------------------------------------------------
2022-09-27T16:34:57.2318446Z Ran 1 test in 0.002s
2022-09-27T16:34:57.2318613Z 
2022-09-27T16:34:57.2318731Z OK (skipped=1)
2022-09-27T16:34:57.2318903Z 
2022-09-27T16:34:57.2319045Z Generating XML reports...
2022-09-27T16:34:57.2354048Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163457.xml
2022-09-27T16:34:57.3591947Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:57.3761184Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.605s)
2022-09-27T16:34:57.3761755Z 
2022-09-27T16:34:57.3762051Z ----------------------------------------------------------------------
2022-09-27T16:34:57.3762394Z Ran 1 test in 1.605s
2022-09-27T16:34:57.3762564Z 
2022-09-27T16:34:57.3762658Z OK (skipped=1)
2022-09-27T16:34:57.3762822Z 
2022-09-27T16:34:57.3762952Z Generating XML reports...
2022-09-27T16:34:57.3796794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163455.xml
2022-09-27T16:34:57.7234998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:57.7235524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:57.7237777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:57.7238281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:58.2417349Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:34:58.2443588Z 
2022-09-27T16:34:58.2443957Z Running tests...
2022-09-27T16:34:58.2446963Z ----------------------------------------------------------------------
2022-09-27T16:34:59.1789210Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:59.1790401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:59.1792563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:59.1793511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:59.5141371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:34:59.5142368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:34:59.5143934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:34:59.5144823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:34:59.6762266Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:34:59.6785184Z 
2022-09-27T16:34:59.6785544Z Running tests...
2022-09-27T16:34:59.6786447Z ----------------------------------------------------------------------
2022-09-27T16:34:59.6808317Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:34:59.6809316Z 
2022-09-27T16:34:59.6809889Z ----------------------------------------------------------------------
2022-09-27T16:34:59.6810552Z Ran 1 test in 0.002s
2022-09-27T16:34:59.6810860Z 
2022-09-27T16:34:59.6811072Z OK (skipped=1)
2022-09-27T16:34:59.6811354Z 
2022-09-27T16:34:59.6811595Z Generating XML reports...
2022-09-27T16:34:59.6842928Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163459.xml
2022-09-27T16:34:59.8520741Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:34:59.8717822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95238
2022-09-27T16:34:59.8722985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95239
2022-09-27T16:35:00.0238429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:35:00.0263452Z 
2022-09-27T16:35:00.0263950Z Running tests...
2022-09-27T16:35:00.0264488Z ----------------------------------------------------------------------
2022-09-27T16:35:01.5616187Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:01.5616840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:01.5619137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:01.5619979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:01.6158801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:01.6159337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:01.6160727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:01.6161214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:01.6816900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:01.6817497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:01.6819946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:01.6820458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:01.7099239Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:01.7298104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95339
2022-09-27T16:35:01.7303612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95340
2022-09-27T16:35:02.0706544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:02.1354608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:02.1355469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:02.1416115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:02.1424225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:02.1813201Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:02.1837072Z 
2022-09-27T16:35:02.1837438Z Running tests...
2022-09-27T16:35:02.1837897Z ----------------------------------------------------------------------
2022-09-27T16:35:02.1859635Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:35:02.1860168Z 
2022-09-27T16:35:02.1860483Z ----------------------------------------------------------------------
2022-09-27T16:35:02.1860835Z Ran 1 test in 0.002s
2022-09-27T16:35:02.1861002Z 
2022-09-27T16:35:02.1861113Z OK (skipped=1)
2022-09-27T16:35:02.1861274Z 
2022-09-27T16:35:02.1861402Z Generating XML reports...
2022-09-27T16:35:02.1896139Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163502.xml
2022-09-27T16:35:02.2370156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:03.4431928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:03.4432868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:03.4433874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:03.4434728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:03.4770645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:03.4771649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:03.4772851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:03.4774173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:03.5506361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5ggmabj
2022-09-27T16:35:03.5507352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5ggmabj/_remote_module_non_scriptable.py
2022-09-27T16:35:03.6784278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprs_60zk
2022-09-27T16:35:03.6785289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprs_60zk/_remote_module_non_scriptable.py
2022-09-27T16:35:04.0173279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:04.0173852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:04.0174654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:04.0175379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:04.0181639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:04.0182384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:04.2288424Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:35:04.2290044Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:35:04.2859381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:04.2859899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:04.2860682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:04.2861174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:04.8163501Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:04.8185449Z 
2022-09-27T16:35:04.8185845Z Running tests...
2022-09-27T16:35:04.8186338Z ----------------------------------------------------------------------
2022-09-27T16:35:04.8207213Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:35:04.8207932Z 
2022-09-27T16:35:04.8208209Z ----------------------------------------------------------------------
2022-09-27T16:35:04.8208545Z Ran 1 test in 0.002s
2022-09-27T16:35:04.8208712Z 
2022-09-27T16:35:04.8208823Z OK (skipped=1)
2022-09-27T16:35:04.8208984Z 
2022-09-27T16:35:04.8209114Z Generating XML reports...
2022-09-27T16:35:04.8242331Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163504.xml
2022-09-27T16:35:04.8833253Z ok (6.639s)
2022-09-27T16:35:04.8833490Z 
2022-09-27T16:35:04.8834096Z ----------------------------------------------------------------------
2022-09-27T16:35:04.8834430Z Ran 1 test in 6.639s
2022-09-27T16:35:04.8834612Z 
2022-09-27T16:35:04.8834706Z OK
2022-09-27T16:35:04.8834843Z 
2022-09-27T16:35:04.8834980Z Generating XML reports...
2022-09-27T16:35:04.8870915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163458.xml
2022-09-27T16:35:05.4470051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hry19gw
2022-09-27T16:35:05.4471010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hry19gw/_remote_module_non_scriptable.py
2022-09-27T16:35:05.4995289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgia611hv
2022-09-27T16:35:05.4996418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgia611hv/_remote_module_non_scriptable.py
2022-09-27T16:35:06.0537589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:06.0538155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:06.6412690Z ok (6.615s)
2022-09-27T16:35:06.6412948Z 
2022-09-27T16:35:06.6413343Z ----------------------------------------------------------------------
2022-09-27T16:35:06.6413689Z Ran 1 test in 6.615s
2022-09-27T16:35:06.6413858Z 
2022-09-27T16:35:06.6413954Z OK
2022-09-27T16:35:06.6414078Z 
2022-09-27T16:35:06.6414222Z Generating XML reports...
2022-09-27T16:35:06.6448375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163500.xml
2022-09-27T16:35:06.9039151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:06.9039688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:06.9040999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:06.9041493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:07.1107828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:07.1108363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:07.1111160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:07.1111659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:07.3907946Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:07.3931869Z 
2022-09-27T16:35:07.3932230Z Running tests...
2022-09-27T16:35:07.3932706Z ----------------------------------------------------------------------
2022-09-27T16:35:07.3954466Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:35:07.3954988Z 
2022-09-27T16:35:07.3955271Z ----------------------------------------------------------------------
2022-09-27T16:35:07.3955608Z Ran 1 test in 0.002s
2022-09-27T16:35:07.3955760Z 
2022-09-27T16:35:07.3955871Z OK (skipped=1)
2022-09-27T16:35:07.3956029Z 
2022-09-27T16:35:07.3956163Z Generating XML reports...
2022-09-27T16:35:07.3991294Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163507.xml
2022-09-27T16:35:07.6122072Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:35:07.6146798Z 
2022-09-27T16:35:07.6147089Z Running tests...
2022-09-27T16:35:07.6147538Z ----------------------------------------------------------------------
2022-09-27T16:35:07.6170276Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-09-27T16:35:07.6170604Z 
2022-09-27T16:35:07.6170884Z ----------------------------------------------------------------------
2022-09-27T16:35:07.6171214Z Ran 1 test in 0.002s
2022-09-27T16:35:07.6171380Z 
2022-09-27T16:35:07.6171490Z OK (skipped=1)
2022-09-27T16:35:07.6171648Z 
2022-09-27T16:35:07.6171775Z Generating XML reports...
2022-09-27T16:35:07.6206535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163507.xml
2022-09-27T16:35:08.7524821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:08.7525375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:08.7527782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:08.7528319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:09.2428316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:35:09.2450626Z 
2022-09-27T16:35:09.2450834Z Running tests...
2022-09-27T16:35:09.2451296Z ----------------------------------------------------------------------
2022-09-27T16:35:09.4248161Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:09.4248788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:09.4249996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:09.4250496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:09.5753073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:09.5753600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:09.5755537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:09.5756055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:09.9333066Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:09.9356366Z 
2022-09-27T16:35:09.9356634Z Running tests...
2022-09-27T16:35:09.9357084Z ----------------------------------------------------------------------
2022-09-27T16:35:09.9379397Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:35:09.9379926Z 
2022-09-27T16:35:09.9380221Z ----------------------------------------------------------------------
2022-09-27T16:35:09.9380562Z Ran 1 test in 0.002s
2022-09-27T16:35:09.9380729Z 
2022-09-27T16:35:09.9380854Z OK (skipped=1)
2022-09-27T16:35:09.9381011Z 
2022-09-27T16:35:09.9381139Z Generating XML reports...
2022-09-27T16:35:09.9415592Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163509.xml
2022-09-27T16:35:10.0688496Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:35:10.0712305Z 
2022-09-27T16:35:10.0712727Z Running tests...
2022-09-27T16:35:10.0713182Z ----------------------------------------------------------------------
2022-09-27T16:35:10.8247439Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:10.8444626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95640
2022-09-27T16:35:10.8449961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95641
2022-09-27T16:35:11.7555580Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:11.7754894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95736
2022-09-27T16:35:11.7760841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95737
2022-09-27T16:35:12.0077143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:12.0077637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:12.0079786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:12.0080690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:12.5776805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:12.5777392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:12.5778573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:12.5779359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:12.6028055Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:12.6052468Z 
2022-09-27T16:35:12.6052815Z Running tests...
2022-09-27T16:35:12.6053262Z ----------------------------------------------------------------------
2022-09-27T16:35:12.6074543Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:35:12.6075171Z 
2022-09-27T16:35:12.6075457Z ----------------------------------------------------------------------
2022-09-27T16:35:12.6075808Z Ran 1 test in 0.002s
2022-09-27T16:35:12.6075957Z 
2022-09-27T16:35:12.6076069Z OK (skipped=1)
2022-09-27T16:35:12.6076229Z 
2022-09-27T16:35:12.6076361Z Generating XML reports...
2022-09-27T16:35:12.6111823Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163512.xml
2022-09-27T16:35:12.6298153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:12.6298892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:12.6300213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:12.6300700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:13.1712140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:13.1712721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:13.1713515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:13.1714221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:13.1820244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:13.2726547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:13.4276980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:13.4277474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:13.4279396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:13.4280494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:13.4301174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:13.4301770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:13.4305927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:13.4306416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:13.9651569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:13.9764173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:13.9765031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:13.9855280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:13.9862061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:14.0780054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:14.5637008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqfqll__p
2022-09-27T16:35:14.5637659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqfqll__p/_remote_module_non_scriptable.py
2022-09-27T16:35:14.6350152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:14.6350661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:14.6353111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:14.6353612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:14.6395195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85scvkjv
2022-09-27T16:35:14.6397798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85scvkjv/_remote_module_non_scriptable.py
2022-09-27T16:35:15.1247758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:15.1248392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:15.1261955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:15.1262547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:15.1770300Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:15.1795217Z 
2022-09-27T16:35:15.1795516Z Running tests...
2022-09-27T16:35:15.1796018Z ----------------------------------------------------------------------
2022-09-27T16:35:15.1817344Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:35:15.1818160Z 
2022-09-27T16:35:15.1818700Z ----------------------------------------------------------------------
2022-09-27T16:35:15.1819025Z Ran 1 test in 0.002s
2022-09-27T16:35:15.1819192Z 
2022-09-27T16:35:15.1819308Z OK (skipped=1)
2022-09-27T16:35:15.1819469Z 
2022-09-27T16:35:15.1819597Z Generating XML reports...
2022-09-27T16:35:15.1856672Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163515.xml
2022-09-27T16:35:15.3449919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwuugc9g
2022-09-27T16:35:15.3450816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwuugc9g/_remote_module_non_scriptable.py
2022-09-27T16:35:15.4339273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfodn6un_
2022-09-27T16:35:15.4339867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfodn6un_/_remote_module_non_scriptable.py
2022-09-27T16:35:15.6567393Z ok (6.411s)
2022-09-27T16:35:15.6567611Z 
2022-09-27T16:35:15.6567994Z ----------------------------------------------------------------------
2022-09-27T16:35:15.6568333Z Ran 1 test in 6.412s
2022-09-27T16:35:15.6568501Z 
2022-09-27T16:35:15.6568579Z OK
2022-09-27T16:35:15.6568716Z 
2022-09-27T16:35:15.6568852Z Generating XML reports...
2022-09-27T16:35:15.6614860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163509.xml
2022-09-27T16:35:17.0442092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:17.0442638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:17.3011419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:17.3011922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:17.3014203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:17.3014703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:17.6899608Z ok (7.618s)
2022-09-27T16:35:17.6899792Z 
2022-09-27T16:35:17.6900175Z ----------------------------------------------------------------------
2022-09-27T16:35:17.6900525Z Ran 1 test in 7.619s
2022-09-27T16:35:17.6900703Z 
2022-09-27T16:35:17.6900783Z OK
2022-09-27T16:35:17.6900925Z 
2022-09-27T16:35:17.6901057Z Generating XML reports...
2022-09-27T16:35:17.6935937Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163510.xml
2022-09-27T16:35:17.8067340Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:17.8091147Z 
2022-09-27T16:35:17.8091439Z Running tests...
2022-09-27T16:35:17.8091890Z ----------------------------------------------------------------------
2022-09-27T16:35:17.8112820Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:35:17.8113578Z 
2022-09-27T16:35:17.8113926Z ----------------------------------------------------------------------
2022-09-27T16:35:17.8114273Z Ran 1 test in 0.002s
2022-09-27T16:35:17.8114425Z 
2022-09-27T16:35:17.8114537Z OK (skipped=1)
2022-09-27T16:35:17.8114693Z 
2022-09-27T16:35:17.8114818Z Generating XML reports...
2022-09-27T16:35:17.8149375Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163517.xml
2022-09-27T16:35:17.8828642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:17.8829237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:17.8830743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:17.8831232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:18.3744880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:35:18.3766917Z 
2022-09-27T16:35:18.3767176Z Running tests...
2022-09-27T16:35:18.3767601Z ----------------------------------------------------------------------
2022-09-27T16:35:19.6947413Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:19.6948304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:19.6949360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:19.6949872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:19.7523207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:19.7523930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:19.7526262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:19.7526738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:19.9413088Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:19.9605648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96006
2022-09-27T16:35:19.9611229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96007
2022-09-27T16:35:20.2038037Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:35:20.2060498Z 
2022-09-27T16:35:20.2060848Z Running tests...
2022-09-27T16:35:20.2061690Z ----------------------------------------------------------------------
2022-09-27T16:35:20.2555140Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:20.2578728Z 
2022-09-27T16:35:20.2579103Z Running tests...
2022-09-27T16:35:20.2579588Z ----------------------------------------------------------------------
2022-09-27T16:35:20.2597273Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:35:20.2597921Z 
2022-09-27T16:35:20.2598525Z ----------------------------------------------------------------------
2022-09-27T16:35:20.2598970Z Ran 1 test in 0.002s
2022-09-27T16:35:20.2599140Z 
2022-09-27T16:35:20.2599250Z OK (skipped=1)
2022-09-27T16:35:20.2599412Z 
2022-09-27T16:35:20.2599540Z Generating XML reports...
2022-09-27T16:35:20.2633731Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163520.xml
2022-09-27T16:35:21.6361706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:21.6362233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:21.6363554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:21.6364048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:21.6364626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:21.6365108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:21.6368120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:21.6368627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:21.8423414Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:21.8615708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96106
2022-09-27T16:35:21.8621516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96107
2022-09-27T16:35:22.1767066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:22.1767670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:22.1768716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:22.1769923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:22.1775130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:22.1776096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:22.2456798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:22.2457431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:22.2458659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:22.2459439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:22.8348580Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:22.8371831Z 
2022-09-27T16:35:22.8372291Z Running tests...
2022-09-27T16:35:22.8373258Z ----------------------------------------------------------------------
2022-09-27T16:35:23.5294254Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:23.5295853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:23.5296565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:23.5297055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:23.5553426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:23.5554093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:23.5555427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:23.5556249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:23.5874921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3nv54al
2022-09-27T16:35:23.5876206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3nv54al/_remote_module_non_scriptable.py
2022-09-27T16:35:23.6028008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73jvqjg8
2022-09-27T16:35:23.6029342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73jvqjg8/_remote_module_non_scriptable.py
2022-09-27T16:35:24.0562049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:24.0562963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:24.0579411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:24.0580467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:24.0710362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:24.0743234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:24.0744306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:24.0758067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:24.0759075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:24.0843697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:24.0845108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:24.0915040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:24.0921589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:24.1859320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:24.5300086Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:24.5508439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96197
2022-09-27T16:35:24.5513689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96198
2022-09-27T16:35:24.6721239Z ok (6.295s)
2022-09-27T16:35:24.6721488Z 
2022-09-27T16:35:24.6721880Z ----------------------------------------------------------------------
2022-09-27T16:35:24.6722209Z Ran 1 test in 6.295s
2022-09-27T16:35:24.6722377Z 
2022-09-27T16:35:24.6722472Z OK
2022-09-27T16:35:24.6722620Z 
2022-09-27T16:35:24.6722754Z Generating XML reports...
2022-09-27T16:35:24.6756003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163518.xml
2022-09-27T16:35:25.5909625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwu0h_7m9
2022-09-27T16:35:25.5910571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwu0h_7m9/_remote_module_non_scriptable.py
2022-09-27T16:35:25.6443441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ezgjnx5
2022-09-27T16:35:25.6444880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ezgjnx5/_remote_module_non_scriptable.py
2022-09-27T16:35:26.2529535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:26.2530057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:26.2531258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:26.2531753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:26.3033318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:26.3033837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:26.3035165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:26.3035632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:26.7668707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:26.7878803Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-09-27T16:35:26.7882165Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-09-27T16:35:26.7895586Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-09-27T16:35:26.7896508Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-09-27T16:35:26.8113525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:26.8114346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:26.8191260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:26.8198631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:26.8776694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:26.8777497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:26.8779085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:26.8779581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:26.9126897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:27.4158384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:35:27.4183013Z 
2022-09-27T16:35:27.4183333Z Running tests...
2022-09-27T16:35:27.4183812Z ----------------------------------------------------------------------
2022-09-27T16:35:27.9752582Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... ok (7.769s)
2022-09-27T16:35:27.9753295Z 
2022-09-27T16:35:27.9753680Z ----------------------------------------------------------------------
2022-09-27T16:35:27.9754004Z Ran 1 test in 7.769s
2022-09-27T16:35:27.9754170Z 
2022-09-27T16:35:27.9754267Z OK
2022-09-27T16:35:27.9754408Z 
2022-09-27T16:35:27.9754535Z Generating XML reports...
2022-09-27T16:35:27.9789352Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163520.xml
2022-09-27T16:35:28.2753242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v9fpgb8
2022-09-27T16:35:28.2754174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v9fpgb8/_remote_module_non_scriptable.py
2022-09-27T16:35:28.3609912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qvh1zdf
2022-09-27T16:35:28.3610754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qvh1zdf/_remote_module_non_scriptable.py
2022-09-27T16:35:28.3865455Z [1664296528.386068] [28fd6574525b:96197:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:35:28.3865976Z [1664296528.386068] [28fd6574525b:96197:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:35:28.3932637Z [1664296528.392890] [28fd6574525b:96198:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:35:28.3933151Z [1664296528.392890] [28fd6574525b:96198:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:35:29.1947485Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:29.2326713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96385
2022-09-27T16:35:29.2332636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96386
2022-09-27T16:35:30.1648934Z ok (7.327s)
2022-09-27T16:35:30.1649157Z 
2022-09-27T16:35:30.1649579Z ----------------------------------------------------------------------
2022-09-27T16:35:30.1649942Z Ran 1 test in 7.328s
2022-09-27T16:35:30.1650108Z 
2022-09-27T16:35:30.1650185Z OK
2022-09-27T16:35:30.1650325Z 
2022-09-27T16:35:30.1650463Z Generating XML reports...
2022-09-27T16:35:30.1686557Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163522.xml
2022-09-27T16:35:30.1880995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:30.1881738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:30.1884181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:30.1884874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:30.6877823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:35:30.6900802Z 
2022-09-27T16:35:30.6901128Z Running tests...
2022-09-27T16:35:30.6901776Z ----------------------------------------------------------------------
2022-09-27T16:35:30.8625474Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:30.8626053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:30.8627382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:30.8627911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:30.9435133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:30.9435629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:30.9437454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:30.9438546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:31.5822509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:31.5823524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:31.5824966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:31.5826263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:31.5832137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:31.5832997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:32.3305125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:32.3305622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:32.3307191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:32.3307761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:32.3785394Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:32.3985123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96497
2022-09-27T16:35:32.3989805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96498
2022-09-27T16:35:32.8511033Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:32.8535298Z 
2022-09-27T16:35:32.8535618Z Running tests...
2022-09-27T16:35:32.8536059Z ----------------------------------------------------------------------
2022-09-27T16:35:32.9111914Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpombhrbr2
2022-09-27T16:35:32.9113405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpombhrbr2/_remote_module_non_scriptable.py
2022-09-27T16:35:32.9488689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k8cvyn4
2022-09-27T16:35:32.9489873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k8cvyn4/_remote_module_non_scriptable.py
2022-09-27T16:35:33.4821199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:33.4821785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:33.4836241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:33.4836776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:33.4982275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:33.4982832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:33.4993324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:33.4993826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:34.0437118Z ok (6.625s)
2022-09-27T16:35:34.0437368Z 
2022-09-27T16:35:34.0437763Z ----------------------------------------------------------------------
2022-09-27T16:35:34.0438110Z Ran 1 test in 6.625s
2022-09-27T16:35:34.0438301Z 
2022-09-27T16:35:34.0438402Z OK
2022-09-27T16:35:34.0438528Z 
2022-09-27T16:35:34.0438681Z Generating XML reports...
2022-09-27T16:35:34.0473624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163527.xml
2022-09-27T16:35:34.0557672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:34.0558157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:34.0560978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:34.0561478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:34.1199053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:34.1199774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:34.1201361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:34.1201849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:34.5936792Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:34.5943979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:34.6135888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96572
2022-09-27T16:35:34.6142316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96573
2022-09-27T16:35:34.6437685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:34.6438765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:34.6452282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:34.6459387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:34.7451926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:36.0947483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9ty00_u
2022-09-27T16:35:36.0948453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9ty00_u/_remote_module_non_scriptable.py
2022-09-27T16:35:36.1994725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuudpavnv
2022-09-27T16:35:36.1995783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuudpavnv/_remote_module_non_scriptable.py
2022-09-27T16:35:36.3036483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:36.3037128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:36.3038344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:36.3038853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:36.3105437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:36.3105915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:36.3109788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:36.3110286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:36.3528457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:36.3528981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:36.3530725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:36.3531225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:36.8268334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:35:36.8292236Z 
2022-09-27T16:35:36.8292554Z Running tests...
2022-09-27T16:35:36.8293030Z ----------------------------------------------------------------------
2022-09-27T16:35:36.8441326Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:36.8676731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:36.8677549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:36.8746407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:36.8753341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:36.9691731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:37.8477131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:37.8477667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:37.8657698Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpa9ty00_u'>
2022-09-27T16:35:37.8658193Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:35:37.8658801Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpuudpavnv'>
2022-09-27T16:35:37.8659246Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:35:38.5121788Z ok (7.822s)
2022-09-27T16:35:38.5122002Z 
2022-09-27T16:35:38.5122404Z ----------------------------------------------------------------------
2022-09-27T16:35:38.5122746Z Ran 1 test in 7.822s
2022-09-27T16:35:38.5122926Z 
2022-09-27T16:35:38.5123006Z OK
2022-09-27T16:35:38.5123144Z 
2022-09-27T16:35:38.5123282Z Generating XML reports...
2022-09-27T16:35:38.5158022Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163530.xml
2022-09-27T16:35:38.5799755Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:38.6011361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96694
2022-09-27T16:35:38.6016541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96695
2022-09-27T16:35:38.8049973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpci2y82sh
2022-09-27T16:35:38.8050823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpci2y82sh/_remote_module_non_scriptable.py
2022-09-27T16:35:38.8395221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvudhaxvy
2022-09-27T16:35:38.8396505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvudhaxvy/_remote_module_non_scriptable.py
2022-09-27T16:35:38.8718268Z [1664296538.871413] [28fd6574525b:96572:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:35:38.8718791Z [1664296538.871413] [28fd6574525b:96572:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:35:38.8750986Z [1664296538.874799] [28fd6574525b:96573:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:35:38.8751681Z [1664296538.874799] [28fd6574525b:96573:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:35:39.2426271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:39.2428294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:39.2551180Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpvudhaxvy'>
2022-09-27T16:35:39.2552557Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:35:39.3588371Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpci2y82sh'>
2022-09-27T16:35:39.3589375Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:35:39.9259391Z ok (7.072s)
2022-09-27T16:35:39.9259732Z 
2022-09-27T16:35:39.9260454Z ----------------------------------------------------------------------
2022-09-27T16:35:39.9261157Z Ran 1 test in 7.072s
2022-09-27T16:35:39.9261360Z 
2022-09-27T16:35:39.9261455Z OK
2022-09-27T16:35:39.9261591Z 
2022-09-27T16:35:39.9261710Z Generating XML reports...
2022-09-27T16:35:39.9297186Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163532.xml
2022-09-27T16:35:40.4069360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:40.4069892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:40.4071371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:40.4071846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:40.4072444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:40.4072903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:40.4076532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:40.4077022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:40.8142393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:40.8142907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:40.8144365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:40.8144851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:40.9498571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:40.9499113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:40.9499902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:40.9500801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:40.9607363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:41.0559084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:41.3773697Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:35:41.3796581Z 
2022-09-27T16:35:41.3797144Z Running tests...
2022-09-27T16:35:41.3797645Z ----------------------------------------------------------------------
2022-09-27T16:35:42.0902429Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:42.0903058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:42.0903908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:42.0904392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:42.3970505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbx9jgm1s
2022-09-27T16:35:42.3971381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbx9jgm1s/_remote_module_non_scriptable.py
2022-09-27T16:35:42.4322323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgrbgkly
2022-09-27T16:35:42.4323697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgrbgkly/_remote_module_non_scriptable.py
2022-09-27T16:35:42.4553844Z <class 'torch.nn.parameter.Parameter'>
2022-09-27T16:35:42.6093352Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:42.6118234Z 
2022-09-27T16:35:42.6118491Z Running tests...
2022-09-27T16:35:42.6118936Z ----------------------------------------------------------------------
2022-09-27T16:35:43.0158817Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... ok (6.186s)
2022-09-27T16:35:43.0159370Z 
2022-09-27T16:35:43.0159782Z ----------------------------------------------------------------------
2022-09-27T16:35:43.0160120Z Ran 1 test in 6.186s
2022-09-27T16:35:43.0160546Z 
2022-09-27T16:35:43.0160646Z OK
2022-09-27T16:35:43.0160789Z 
2022-09-27T16:35:43.0160917Z Generating XML reports...
2022-09-27T16:35:43.0195109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163536.xml
2022-09-27T16:35:43.0428004Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:43.0634757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96852
2022-09-27T16:35:43.0640553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96853
2022-09-27T16:35:44.3028333Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:44.3231884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96949
2022-09-27T16:35:44.3237118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96950
2022-09-27T16:35:44.7294140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:44.7294659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:44.7296298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:44.7296957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:44.7422716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:44.7423189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:44.7427082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:44.7427836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:45.2920634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:45.2921193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:45.2922139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:45.2922626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:45.3126353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:45.3393904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:45.3394728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:45.3431355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:45.3438935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:45.4408299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:45.7953759Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:35:45.7976395Z 
2022-09-27T16:35:45.7976922Z Running tests...
2022-09-27T16:35:45.7977412Z ----------------------------------------------------------------------
2022-09-27T16:35:46.0452253Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:46.0453134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:46.0454194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:46.0454716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:46.0623790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:46.0624304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:46.0625838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:46.0626325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:46.5833490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:46.6164457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:46.6165295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:46.6241295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:46.6248207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:46.6354646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38ebgpsd
2022-09-27T16:35:46.6355192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75udmot7
2022-09-27T16:35:46.6358686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75udmot7/_remote_module_non_scriptable.py
2022-09-27T16:35:46.6359264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38ebgpsd/_remote_module_non_scriptable.py
2022-09-27T16:35:46.7177888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:46.7289444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4l2y8gd
2022-09-27T16:35:46.7290644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4l2y8gd/_remote_module_non_scriptable.py
2022-09-27T16:35:46.8322774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrxj5l60
2022-09-27T16:35:46.8323928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrxj5l60/_remote_module_non_scriptable.py
2022-09-27T16:35:47.4963074Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:47.5171462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97035
2022-09-27T16:35:47.5176624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97036
2022-09-27T16:35:48.1531982Z [1664296548.151727] [28fd6574525b:96950:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:35:48.1533291Z [1664296548.151727] [28fd6574525b:96950:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:35:48.2212337Z [1664296548.220579] [28fd6574525b:96949:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:35:48.2214314Z [1664296548.220579] [28fd6574525b:96949:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:35:48.4018044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:48.4019066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:48.4029200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:48.4029935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:48.5630358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:48.5631389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:49.2331896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:49.2332418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:49.2334413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:49.2334908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:49.2566283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:49.2566762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:49.2570362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:49.2570851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:49.2774137Z ok (7.898s)
2022-09-27T16:35:49.2774357Z 
2022-09-27T16:35:49.2774893Z ----------------------------------------------------------------------
2022-09-27T16:35:49.2775513Z Ran 1 test in 7.898s
2022-09-27T16:35:49.2775814Z 
2022-09-27T16:35:49.2775976Z OK
2022-09-27T16:35:49.2776227Z 
2022-09-27T16:35:49.2776406Z Generating XML reports...
2022-09-27T16:35:49.2813915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163541.xml
2022-09-27T16:35:49.3352030Z ok (6.723s)
2022-09-27T16:35:49.3352425Z 
2022-09-27T16:35:49.3353086Z ----------------------------------------------------------------------
2022-09-27T16:35:49.3353727Z Ran 1 test in 6.723s
2022-09-27T16:35:49.3354044Z 
2022-09-27T16:35:49.3354216Z OK
2022-09-27T16:35:49.3354473Z 
2022-09-27T16:35:49.3354719Z Generating XML reports...
2022-09-27T16:35:49.3389424Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163542.xml
2022-09-27T16:35:49.8379483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:49.8380080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:49.8380874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:49.8381583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:49.8487751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:49.9394203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:51.2432957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vbmxy7j
2022-09-27T16:35:51.2437798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vbmxy7j/_remote_module_non_scriptable.py
2022-09-27T16:35:51.2936349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnrve881
2022-09-27T16:35:51.2937002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnrve881/_remote_module_non_scriptable.py
2022-09-27T16:35:51.4924962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:51.4925493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:51.4927997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:51.4928483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:51.5633676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:51.5634181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:51.5636631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:51.5637109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:51.8287083Z ok (6.031s)
2022-09-27T16:35:51.8287471Z 
2022-09-27T16:35:51.8288240Z ----------------------------------------------------------------------
2022-09-27T16:35:51.8288713Z Ran 1 test in 6.031s
2022-09-27T16:35:51.8288895Z 
2022-09-27T16:35:51.8288989Z OK
2022-09-27T16:35:51.8289129Z 
2022-09-27T16:35:51.8289247Z Generating XML reports...
2022-09-27T16:35:51.8345121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163545.xml
2022-09-27T16:35:52.0632073Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:35:52.0654931Z 
2022-09-27T16:35:52.0655362Z Running tests...
2022-09-27T16:35:52.0656338Z ----------------------------------------------------------------------
2022-09-27T16:35:52.0689676Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:35:52.0714275Z 
2022-09-27T16:35:52.0714634Z Running tests...
2022-09-27T16:35:52.0715580Z ----------------------------------------------------------------------
2022-09-27T16:35:53.7680696Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:53.7874408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97260
2022-09-27T16:35:53.7881161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97261
2022-09-27T16:35:53.7920088Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:53.8122582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97263
2022-09-27T16:35:53.8128476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97264
2022-09-27T16:35:53.9448810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:53.9449475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:53.9450973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:53.9451782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:54.4759248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:35:54.4808179Z 
2022-09-27T16:35:54.4814231Z Running tests...
2022-09-27T16:35:54.4821458Z ----------------------------------------------------------------------
2022-09-27T16:35:55.4662614Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:55.4663236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:55.4664049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:55.4664853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:55.4725052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:55.4725533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:55.4729249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:55.4729727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:55.5066786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:55.5067344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:55.5068731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:55.5069206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:55.5285300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:55.5285796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:55.5287882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:55.5288376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:55.9985204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:56.0243732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:56.0244265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:56.0245316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:56.0289361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:56.0296785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:56.0659747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:56.0660522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:56.0751373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:56.0758357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:56.0761756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:35:56.0931127Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:35:56.1132003Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97405
2022-09-27T16:35:56.1138639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97406
2022-09-27T16:35:56.1257850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:56.1673524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:56.1676882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:35:56.1677807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:35:56.1777497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:35:57.4713374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5jyqnws
2022-09-27T16:35:57.4714041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5jyqnws/_remote_module_non_scriptable.py
2022-09-27T16:35:57.5292827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbltnuqe
2022-09-27T16:35:57.5293907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbltnuqe/_remote_module_non_scriptable.py
2022-09-27T16:35:57.5598610Z [1664296557.559393] [28fd6574525b:97260:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:35:57.5599161Z [1664296557.559393] [28fd6574525b:97260:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:35:57.5670033Z [1664296557.566640] [28fd6574525b:97261:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:35:57.5670848Z [1664296557.566640] [28fd6574525b:97261:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:35:57.6025237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71_87t9k
2022-09-27T16:35:57.6026941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71_87t9k/_remote_module_non_scriptable.py
2022-09-27T16:35:57.6177109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc_bfuzp
2022-09-27T16:35:57.6178484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc_bfuzp/_remote_module_non_scriptable.py
2022-09-27T16:35:57.7921820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:57.7922391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:57.7923644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:57.7924133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:57.8134623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:35:57.8135151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:35:57.8136868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:35:57.8137361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:35:58.3737990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:58.3738592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:58.3820795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:35:58.3821354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:35:58.3822156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:58.3822906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:35:58.3829607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:35:58.3830100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:35:59.0004594Z ok (6.935s)
2022-09-27T16:35:59.0004863Z 
2022-09-27T16:35:59.0005264Z ----------------------------------------------------------------------
2022-09-27T16:35:59.0005606Z Ran 1 test in 6.935s
2022-09-27T16:35:59.0005775Z 
2022-09-27T16:35:59.0005893Z OK
2022-09-27T16:35:59.0006012Z 
2022-09-27T16:35:59.0006156Z Generating XML reports...
2022-09-27T16:35:59.0041890Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163552.xml
2022-09-27T16:35:59.3389913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:59.3390515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:59.3401314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:59.3401813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:35:59.8717142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljqtm3y5
2022-09-27T16:35:59.8717756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljqtm3y5/_remote_module_non_scriptable.py
2022-09-27T16:35:59.8747556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1q14i5a
2022-09-27T16:35:59.8750295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1q14i5a/_remote_module_non_scriptable.py
2022-09-27T16:36:00.2270731Z ok (8.155s)
2022-09-27T16:36:00.2271091Z 
2022-09-27T16:36:00.2271489Z ----------------------------------------------------------------------
2022-09-27T16:36:00.2271837Z Ran 1 test in 8.155s
2022-09-27T16:36:00.2271987Z 
2022-09-27T16:36:00.2272083Z OK
2022-09-27T16:36:00.2272220Z 
2022-09-27T16:36:00.2272353Z Generating XML reports...
2022-09-27T16:36:00.2315956Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163552.xml
2022-09-27T16:36:00.4931847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:00.4932889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:01.0259261Z ok (6.543s)
2022-09-27T16:36:01.0259705Z 
2022-09-27T16:36:01.0260387Z ----------------------------------------------------------------------
2022-09-27T16:36:01.0261030Z Ran 1 test in 6.544s
2022-09-27T16:36:01.0261361Z 
2022-09-27T16:36:01.0261508Z OK
2022-09-27T16:36:01.0261765Z 
2022-09-27T16:36:01.0262013Z Generating XML reports...
2022-09-27T16:36:01.0296166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163554.xml
2022-09-27T16:36:01.2563103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:01.2564077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:01.2565928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:01.2566873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:01.7550879Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:01.7574749Z 
2022-09-27T16:36:01.7575334Z Running tests...
2022-09-27T16:36:01.7576183Z ----------------------------------------------------------------------
2022-09-27T16:36:01.7599393Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:36:01.7600112Z 
2022-09-27T16:36:01.7600992Z ----------------------------------------------------------------------
2022-09-27T16:36:01.7601642Z Ran 1 test in 0.003s
2022-09-27T16:36:01.7601959Z 
2022-09-27T16:36:01.7602138Z OK (skipped=1)
2022-09-27T16:36:01.7602453Z 
2022-09-27T16:36:01.7602698Z Generating XML reports...
2022-09-27T16:36:01.7637803Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163601.xml
2022-09-27T16:36:02.5432128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:02.5432628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:02.5434956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:02.5435751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:03.0262944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:03.0284286Z 
2022-09-27T16:36:03.0284522Z Running tests...
2022-09-27T16:36:03.0284980Z ----------------------------------------------------------------------
2022-09-27T16:36:03.1954034Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:03.1955199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:03.1956428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:03.1956941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:03.6990687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:03.6991198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:03.6993451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:03.6993953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:03.7065446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:36:03.7091091Z 
2022-09-27T16:36:03.7091349Z Running tests...
2022-09-27T16:36:03.7091785Z ----------------------------------------------------------------------
2022-09-27T16:36:04.2054732Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:04.2076284Z 
2022-09-27T16:36:04.2076779Z Running tests...
2022-09-27T16:36:04.2077281Z ----------------------------------------------------------------------
2022-09-27T16:36:04.2098782Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:36:04.2099143Z 
2022-09-27T16:36:04.2099414Z ----------------------------------------------------------------------
2022-09-27T16:36:04.2099735Z Ran 1 test in 0.002s
2022-09-27T16:36:04.2099906Z 
2022-09-27T16:36:04.2100014Z OK (skipped=1)
2022-09-27T16:36:04.2100167Z 
2022-09-27T16:36:04.2100291Z Generating XML reports...
2022-09-27T16:36:04.2134832Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163604.xml
2022-09-27T16:36:04.5938203Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:04.6474584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97653
2022-09-27T16:36:04.6479373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97654
2022-09-27T16:36:05.3741695Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:05.4207914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97749
2022-09-27T16:36:05.4257117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97750
2022-09-27T16:36:06.2692369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:06.2693367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:06.2694555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:06.2695558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:06.4226061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:06.4226910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:06.4228065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:06.4228558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:06.4569661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:06.4570293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:06.4572560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:06.4573064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:06.7567767Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:06.7589798Z 
2022-09-27T16:36:06.7590064Z Running tests...
2022-09-27T16:36:06.7590526Z ----------------------------------------------------------------------
2022-09-27T16:36:06.7610054Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:36:06.7610535Z 
2022-09-27T16:36:06.7610971Z ----------------------------------------------------------------------
2022-09-27T16:36:06.7611291Z Ran 1 test in 0.002s
2022-09-27T16:36:06.7611459Z 
2022-09-27T16:36:06.7611571Z OK (skipped=1)
2022-09-27T16:36:06.7611730Z 
2022-09-27T16:36:06.7611859Z Generating XML reports...
2022-09-27T16:36:06.7644744Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163606.xml
2022-09-27T16:36:06.9267741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:06.9694877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:06.9695715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:06.9775357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:06.9782018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:06.9784645Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:36:07.0552542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:07.0553060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:07.0555575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:07.0556105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:07.0706139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:07.0707822Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:36:07.1018206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:07.1018691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:07.1020042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:07.1020528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:07.6570353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:07.6571416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:07.6572924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:07.6574323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:07.6579253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:07.6580929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:08.4289704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgr6vovq
2022-09-27T16:36:08.4290353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgr6vovq/_remote_module_non_scriptable.py
2022-09-27T16:36:08.5148806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3emq8vs
2022-09-27T16:36:08.5149753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3emq8vs/_remote_module_non_scriptable.py
2022-09-27T16:36:08.7291408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:08.7291931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:08.7293165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:08.7293655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:09.1344148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktl7s9ok
2022-09-27T16:36:09.1344768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktl7s9ok/_remote_module_non_scriptable.py
2022-09-27T16:36:09.1459348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxyz41nt
2022-09-27T16:36:09.1462340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxyz41nt/_remote_module_non_scriptable.py
2022-09-27T16:36:09.2727716Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:09.2751069Z 
2022-09-27T16:36:09.2751309Z Running tests...
2022-09-27T16:36:09.2752030Z ----------------------------------------------------------------------
2022-09-27T16:36:09.2771272Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:36:09.2771728Z 
2022-09-27T16:36:09.2772009Z ----------------------------------------------------------------------
2022-09-27T16:36:09.2772346Z Ran 1 test in 0.002s
2022-09-27T16:36:09.2772650Z 
2022-09-27T16:36:09.2772844Z OK (skipped=1)
2022-09-27T16:36:09.2773016Z 
2022-09-27T16:36:09.2773455Z Generating XML reports...
2022-09-27T16:36:09.2807626Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163609.xml
2022-09-27T16:36:09.6208675Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:36:10.0150112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.0151124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.0161808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.0162525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.0370236Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:36:10.0370861Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:36:10.2363897Z ok (6.527s)
2022-09-27T16:36:10.2364242Z 
2022-09-27T16:36:10.2364638Z ----------------------------------------------------------------------
2022-09-27T16:36:10.2364984Z Ran 1 test in 6.527s
2022-09-27T16:36:10.2365153Z 
2022-09-27T16:36:10.2365249Z OK
2022-09-27T16:36:10.2365383Z 
2022-09-27T16:36:10.2365504Z Generating XML reports...
2022-09-27T16:36:10.2401945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163603.xml
2022-09-27T16:36:10.2904524Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:36:10.2905875Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:36:10.3010029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.3012016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.3030426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.3034800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.3404235Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:36:10.3405573Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:36:10.5009229Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:36:10.5011920Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:36:10.5073417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.5073934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.5086873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:10.5087380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:11.3053700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:11.3054367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:11.3056375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:11.3056898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:11.4642725Z ok (8.436s)
2022-09-27T16:36:11.4642932Z 
2022-09-27T16:36:11.4643291Z ----------------------------------------------------------------------
2022-09-27T16:36:11.4643637Z Ran 1 test in 8.436s
2022-09-27T16:36:11.4643808Z 
2022-09-27T16:36:11.4643880Z OK
2022-09-27T16:36:11.4644015Z 
2022-09-27T16:36:11.4644150Z Generating XML reports...
2022-09-27T16:36:11.4679874Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163603.xml
2022-09-27T16:36:11.8670565Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:11.8693318Z 
2022-09-27T16:36:11.8693535Z Running tests...
2022-09-27T16:36:11.8693962Z ----------------------------------------------------------------------
2022-09-27T16:36:11.8721027Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-09-27T16:36:11.8721355Z 
2022-09-27T16:36:11.8721953Z ----------------------------------------------------------------------
2022-09-27T16:36:11.8722287Z Ran 1 test in 0.003s
2022-09-27T16:36:11.8722433Z 
2022-09-27T16:36:11.8722544Z OK (skipped=1)
2022-09-27T16:36:11.8722699Z 
2022-09-27T16:36:11.8722827Z Generating XML reports...
2022-09-27T16:36:11.8757375Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163611.xml
2022-09-27T16:36:12.4040998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:12.4041642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:12.4042880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:12.4043363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:12.9635719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:36:12.9660733Z 
2022-09-27T16:36:12.9661172Z Running tests...
2022-09-27T16:36:12.9661688Z ----------------------------------------------------------------------
2022-09-27T16:36:13.5447135Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:13.5447764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:13.5448624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:13.5449127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:13.7847802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:13.7848398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:13.7850542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:13.7851042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:14.0420560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:14.0442418Z 
2022-09-27T16:36:14.0442873Z Running tests...
2022-09-27T16:36:14.0443385Z ----------------------------------------------------------------------
2022-09-27T16:36:14.2873739Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:14.2896833Z 
2022-09-27T16:36:14.2897281Z Running tests...
2022-09-27T16:36:14.2898091Z ----------------------------------------------------------------------
2022-09-27T16:36:14.5722036Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:14.5929256Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98021
2022-09-27T16:36:14.5934341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98022
2022-09-27T16:36:15.7380816Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:15.7576409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98086
2022-09-27T16:36:15.7582025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98087
2022-09-27T16:36:15.9747284Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:15.9949125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98089
2022-09-27T16:36:15.9954592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98090
2022-09-27T16:36:16.2785536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:16.2786414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:16.2787941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:16.2788426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:16.3407817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:16.3408667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:16.3409639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:16.3410126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:16.9610324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:16.9610872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:16.9611672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:16.9612382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:16.9717862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:17.0624381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:17.4332379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:17.4332907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:17.4334213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:17.4334709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:17.4941790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:17.4942293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:17.4944675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:17.4945174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:17.7074735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:17.7075258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:17.7077698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:17.7078213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:17.7612324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:17.7612863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:17.7615155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:17.7615647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:17.9599952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:18.0337731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:18.0338580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:18.0412808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:18.0419843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:18.0422611Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:36:18.1352327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:18.1354089Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:36:18.2372093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:18.3066226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:18.3067068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:18.3084408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:18.3092905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:18.3586143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj8ppawv
2022-09-27T16:36:18.3587368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj8ppawv/_remote_module_non_scriptable.py
2022-09-27T16:36:18.4080763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:18.4734749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0r3o7t11
2022-09-27T16:36:18.4735642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0r3o7t11/_remote_module_non_scriptable.py
2022-09-27T16:36:18.9371978Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:36:18.9373930Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:36:19.4679375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9k3w0yk
2022-09-27T16:36:19.4679998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9k3w0yk/_remote_module_non_scriptable.py
2022-09-27T16:36:19.5753918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52qtkk70
2022-09-27T16:36:19.5755239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52qtkk70/_remote_module_non_scriptable.py
2022-09-27T16:36:19.7051014Z ok (6.739s)
2022-09-27T16:36:19.7051211Z 
2022-09-27T16:36:19.7051624Z ----------------------------------------------------------------------
2022-09-27T16:36:19.7051964Z Ran 1 test in 6.739s
2022-09-27T16:36:19.7052402Z 
2022-09-27T16:36:19.7052480Z OK
2022-09-27T16:36:19.7052615Z 
2022-09-27T16:36:19.7052754Z Generating XML reports...
2022-09-27T16:36:19.7091323Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163612.xml
2022-09-27T16:36:19.7484304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvxzupxz
2022-09-27T16:36:19.7486425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvxzupxz/_remote_module_non_scriptable.py
2022-09-27T16:36:19.8627785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasdcc8cc
2022-09-27T16:36:19.8628659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasdcc8cc/_remote_module_non_scriptable.py
2022-09-27T16:36:19.8905141Z [1664296579.889962] [28fd6574525b:98089:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:36:19.8905687Z [1664296579.889962] [28fd6574525b:98089:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:36:19.8950580Z [1664296579.894719] [28fd6574525b:98090:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:36:19.8951077Z [1664296579.894719] [28fd6574525b:98090:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:36:20.6895543Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:36:20.6974033Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:36:20.7050591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:20.7051107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.2161657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.2162259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.2173702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.2174431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.2179984Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:36:21.2180568Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:36:21.2208037Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:36:21.2208704Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:36:21.2209369Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:36:21.2211357Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:36:21.2212263Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:36:21.2213153Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:36:21.3068668Z ok (7.017s)
2022-09-27T16:36:21.3068871Z 
2022-09-27T16:36:21.3069277Z ----------------------------------------------------------------------
2022-09-27T16:36:21.3069616Z Ran 1 test in 7.017s
2022-09-27T16:36:21.3069771Z 
2022-09-27T16:36:21.3069869Z OK
2022-09-27T16:36:21.3070007Z 
2022-09-27T16:36:21.3070158Z Generating XML reports...
2022-09-27T16:36:21.3104999Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163614.xml
2022-09-27T16:36:21.5731865Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:36:21.5734024Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:36:21.5795109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.5796085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.5808011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.5808991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:21.5814112Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:36:21.5814836Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:36:21.5841530Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:36:21.5842186Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:36:21.5843838Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:36:21.5844530Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:36:21.8639879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:21.8640655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:21.8641841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:21.8642338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:22.4443362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:36:22.4480291Z 
2022-09-27T16:36:22.4481067Z Running tests...
2022-09-27T16:36:22.4481820Z ----------------------------------------------------------------------
2022-09-27T16:36:22.5727631Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... ok (8.528s)
2022-09-27T16:36:22.5728403Z 
2022-09-27T16:36:22.5728785Z ----------------------------------------------------------------------
2022-09-27T16:36:22.5729107Z Ran 1 test in 8.528s
2022-09-27T16:36:22.5729271Z 
2022-09-27T16:36:22.5729369Z OK
2022-09-27T16:36:22.5729503Z 
2022-09-27T16:36:22.5729632Z Generating XML reports...
2022-09-27T16:36:22.5764055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163614.xml
2022-09-27T16:36:23.3807529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:23.3808044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:23.3810425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:23.3810920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:23.9377700Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:23.9399101Z 
2022-09-27T16:36:23.9399584Z Running tests...
2022-09-27T16:36:23.9400095Z ----------------------------------------------------------------------
2022-09-27T16:36:24.1489273Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:24.1689291Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98373
2022-09-27T16:36:24.1694345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98374
2022-09-27T16:36:24.7064131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:24.7064685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:24.7067609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:24.7068113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:25.3031958Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:25.3054391Z 
2022-09-27T16:36:25.3055024Z Running tests...
2022-09-27T16:36:25.3055967Z ----------------------------------------------------------------------
2022-09-27T16:36:25.5677938Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:25.5878017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98440
2022-09-27T16:36:25.5883620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98441
2022-09-27T16:36:25.8134101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:25.8135104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:25.8136243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:25.8137209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:25.8800043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:25.8801373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:25.8802558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:25.8803481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:26.4839214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:26.4839814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:26.4841104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:26.4842099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:26.4949215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:26.5056142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7v6qf_js
2022-09-27T16:36:26.5058273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7v6qf_js/_remote_module_non_scriptable.py
2022-09-27T16:36:26.5853916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:26.5953755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfc07ugp
2022-09-27T16:36:26.5956323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfc07ugp/_remote_module_non_scriptable.py
2022-09-27T16:36:26.6127521Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6130558Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6133151Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T16:36:26.6134298Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:36:26.6137246Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1061.)
2022-09-27T16:36:26.6139154Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-09-27T16:36:26.6140106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:26.6141043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:26.6142301Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6145006Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6150459Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6152374Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6158708Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6161960Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6166258Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6168688Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6173787Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.6176349Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-09-27T16:36:26.9537587Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:26.9738558Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98519
2022-09-27T16:36:26.9744113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98520
2022-09-27T16:36:26.9773489Z ok (4.529s)
2022-09-27T16:36:26.9773899Z 
2022-09-27T16:36:26.9774642Z ----------------------------------------------------------------------
2022-09-27T16:36:26.9775050Z Ran 1 test in 4.529s
2022-09-27T16:36:26.9775229Z 
2022-09-27T16:36:26.9775307Z OK
2022-09-27T16:36:26.9775447Z 
2022-09-27T16:36:26.9775580Z Generating XML reports...
2022-09-27T16:36:26.9830574Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163622.xml
2022-09-27T16:36:27.1868651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:27.1869474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:27.1871512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:27.1872009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:27.2572546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:27.2573057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:27.2574294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:27.2574819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:27.7445409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:27.8573690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:27.8575338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:27.8660683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:27.8667846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:27.9589199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:28.6655486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:28.6656498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:28.6658274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:28.6659265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:28.6884792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:28.6885790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:28.6886973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:28.6887926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:29.1666755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:29.1668124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:29.1669316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:29.1670290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:29.2203946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:29.2270083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:29.2271574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:29.2305366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:29.2312541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:29.2315580Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:36:29.2344185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupvgmrun
2022-09-27T16:36:29.2347134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupvgmrun/_remote_module_non_scriptable.py
2022-09-27T16:36:29.2871490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj74im333
2022-09-27T16:36:29.2872648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj74im333/_remote_module_non_scriptable.py
2022-09-27T16:36:29.3130832Z [1664296589.312654] [28fd6574525b:98440:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:36:29.3131597Z [1664296589.312654] [28fd6574525b:98440:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:36:29.3212977Z [1664296589.320964] [28fd6574525b:98441:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:36:29.3213607Z [1664296589.320964] [28fd6574525b:98441:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:36:29.3281752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:29.3284604Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:36:29.6682138Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:36:29.6682952Z   warnings.warn(
2022-09-27T16:36:29.6684048Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:36:29.6684788Z   warnings.warn(
2022-09-27T16:36:29.6820445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:36:29.6845556Z 
2022-09-27T16:36:29.6846168Z Running tests...
2022-09-27T16:36:29.6846943Z ----------------------------------------------------------------------
2022-09-27T16:36:30.6184345Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75z1viss
2022-09-27T16:36:30.6185722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75z1viss/_remote_module_non_scriptable.py
2022-09-27T16:36:30.7037121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa04s9_79
2022-09-27T16:36:30.7038214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa04s9_79/_remote_module_non_scriptable.py
2022-09-27T16:36:30.8018482Z ok (6.862s)
2022-09-27T16:36:30.8018706Z 
2022-09-27T16:36:30.8019098Z ----------------------------------------------------------------------
2022-09-27T16:36:30.8019429Z Ran 1 test in 6.862s
2022-09-27T16:36:30.8019598Z 
2022-09-27T16:36:30.8019699Z OK
2022-09-27T16:36:30.8019836Z 
2022-09-27T16:36:30.8019980Z Generating XML reports...
2022-09-27T16:36:30.8075394Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163623.xml
2022-09-27T16:36:31.4325864Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:31.4527724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98651
2022-09-27T16:36:31.4532945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98652
2022-09-27T16:36:32.4286943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:32.4287521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:32.4315750Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-09-27T16:36:32.4316868Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-09-27T16:36:32.4328817Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:36:32.4329440Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:36:32.4331755Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:36:32.4343264Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:36:32.4344264Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:36:32.4344939Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:36:32.4894556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:32.5141255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:33.1316221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:33.1316743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:33.1319304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:33.1319805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:33.1623092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:33.1623572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:33.1627325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:33.1627855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:33.1885508Z ok (7.883s)
2022-09-27T16:36:33.1885695Z 
2022-09-27T16:36:33.1885997Z ----------------------------------------------------------------------
2022-09-27T16:36:33.1886326Z Ran 1 test in 7.883s
2022-09-27T16:36:33.1886494Z 
2022-09-27T16:36:33.1886590Z OK
2022-09-27T16:36:33.1886727Z 
2022-09-27T16:36:33.1886857Z Generating XML reports...
2022-09-27T16:36:33.1923743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163625.xml
2022-09-27T16:36:33.2419737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:33.2420228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:33.2421446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:33.2421941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:33.6603418Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:33.6626976Z 
2022-09-27T16:36:33.6627133Z Running tests...
2022-09-27T16:36:33.6627901Z ----------------------------------------------------------------------
2022-09-27T16:36:33.6651437Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:36:33.6651775Z 
2022-09-27T16:36:33.6652220Z ----------------------------------------------------------------------
2022-09-27T16:36:33.6652949Z Ran 1 test in 0.002s
2022-09-27T16:36:33.6653316Z 
2022-09-27T16:36:33.6653517Z OK (skipped=1)
2022-09-27T16:36:33.6653753Z 
2022-09-27T16:36:33.6653887Z Generating XML reports...
2022-09-27T16:36:33.6695917Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163633.xml
2022-09-27T16:36:33.7865369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:33.7866481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:33.7867289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:33.7868005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:33.7974656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:33.8880995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:35.2635010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2qh_gpk
2022-09-27T16:36:35.2635762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2qh_gpk/_remote_module_non_scriptable.py
2022-09-27T16:36:35.3126645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyz0z9iqc
2022-09-27T16:36:35.3127978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyz0z9iqc/_remote_module_non_scriptable.py
2022-09-27T16:36:35.3357955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:35.3358472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:35.3362039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:35.3362553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:35.6479028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:35.6480027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:35.6480927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:35.6481761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:35.7501643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:35.7502178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:35.8518347Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:35.8543174Z 
2022-09-27T16:36:35.8543526Z Running tests...
2022-09-27T16:36:35.8544012Z ----------------------------------------------------------------------
2022-09-27T16:36:35.8565948Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:35.8566405Z 
2022-09-27T16:36:35.8566710Z ----------------------------------------------------------------------
2022-09-27T16:36:35.8567502Z Ran 1 test in 0.002s
2022-09-27T16:36:35.8567672Z 
2022-09-27T16:36:35.8567785Z OK (skipped=1)
2022-09-27T16:36:35.8567943Z 
2022-09-27T16:36:35.8568080Z Generating XML reports...
2022-09-27T16:36:35.8602509Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163635.xml
2022-09-27T16:36:36.1544861Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:36.1568013Z 
2022-09-27T16:36:36.1568281Z Running tests...
2022-09-27T16:36:36.1568746Z ----------------------------------------------------------------------
2022-09-27T16:36:36.1590639Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:36:36.1591052Z 
2022-09-27T16:36:36.1591373Z ----------------------------------------------------------------------
2022-09-27T16:36:36.1591939Z Ran 1 test in 0.002s
2022-09-27T16:36:36.1592111Z 
2022-09-27T16:36:36.1592234Z OK (skipped=1)
2022-09-27T16:36:36.1592392Z 
2022-09-27T16:36:36.1592503Z Generating XML reports...
2022-09-27T16:36:36.1626719Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163636.xml
2022-09-27T16:36:36.3647737Z ok (6.680s)
2022-09-27T16:36:36.3648085Z 
2022-09-27T16:36:36.3648722Z ----------------------------------------------------------------------
2022-09-27T16:36:36.3649335Z Ran 1 test in 6.680s
2022-09-27T16:36:36.3649660Z 
2022-09-27T16:36:36.3649830Z OK
2022-09-27T16:36:36.3650081Z 
2022-09-27T16:36:36.3650330Z Generating XML reports...
2022-09-27T16:36:36.3684851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163629.xml
2022-09-27T16:36:37.9695979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:37.9696514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:37.9698707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:37.9699185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:38.2353678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:38.2354188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:38.2356273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:38.2356772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:38.4628759Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:38.4651970Z 
2022-09-27T16:36:38.4652407Z Running tests...
2022-09-27T16:36:38.4653020Z ----------------------------------------------------------------------
2022-09-27T16:36:38.4673021Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:38.4673467Z 
2022-09-27T16:36:38.4673762Z ----------------------------------------------------------------------
2022-09-27T16:36:38.4674082Z Ran 1 test in 0.002s
2022-09-27T16:36:38.4674261Z 
2022-09-27T16:36:38.4674373Z OK (skipped=1)
2022-09-27T16:36:38.4674531Z 
2022-09-27T16:36:38.4674657Z Generating XML reports...
2022-09-27T16:36:38.4709031Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163638.xml
2022-09-27T16:36:38.5738694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:38.5739194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:38.5740735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:38.5741223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:38.7327920Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:38.7350956Z 
2022-09-27T16:36:38.7351161Z Running tests...
2022-09-27T16:36:38.7351594Z ----------------------------------------------------------------------
2022-09-27T16:36:39.0672120Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:36:39.0694757Z 
2022-09-27T16:36:39.0694953Z Running tests...
2022-09-27T16:36:39.0695409Z ----------------------------------------------------------------------
2022-09-27T16:36:40.3560331Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:40.3758958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98972
2022-09-27T16:36:40.3764489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98973
2022-09-27T16:36:40.4134387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:40.4134891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:40.4136584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:40.4137067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:40.6766633Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:40.7073870Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98976
2022-09-27T16:36:40.7079589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98977
2022-09-27T16:36:40.9193830Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:40.9217387Z 
2022-09-27T16:36:40.9217775Z Running tests...
2022-09-27T16:36:40.9218680Z ----------------------------------------------------------------------
2022-09-27T16:36:40.9240570Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:40.9241241Z 
2022-09-27T16:36:40.9241538Z ----------------------------------------------------------------------
2022-09-27T16:36:40.9241859Z Ran 1 test in 0.002s
2022-09-27T16:36:40.9242025Z 
2022-09-27T16:36:40.9242136Z OK (skipped=1)
2022-09-27T16:36:40.9242295Z 
2022-09-27T16:36:40.9242714Z Generating XML reports...
2022-09-27T16:36:40.9276549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163640.xml
2022-09-27T16:36:42.0759141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:42.0760813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:42.0762203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:42.0763298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:42.1300544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:42.1301056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:42.1302091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:42.1302585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:42.4395824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:42.4396800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:42.4397991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:42.4398952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:42.4978201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:42.4979151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:42.4980802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:42.4981765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:42.5799176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:42.6373957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:42.6374847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:42.6407153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:42.6413989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:42.7387011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:42.9548529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:42.9549536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:42.9550678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:42.9551617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:43.0436998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:43.0438148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:43.0439663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:43.0441353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:43.0549135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:43.1452122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:43.4743488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:43.4766724Z 
2022-09-27T16:36:43.4767400Z Running tests...
2022-09-27T16:36:43.4767914Z ----------------------------------------------------------------------
2022-09-27T16:36:43.4789171Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:43.4789993Z 
2022-09-27T16:36:43.4790275Z ----------------------------------------------------------------------
2022-09-27T16:36:43.4790611Z Ran 1 test in 0.002s
2022-09-27T16:36:43.4790779Z 
2022-09-27T16:36:43.4790890Z OK (skipped=1)
2022-09-27T16:36:43.4791046Z 
2022-09-27T16:36:43.4791184Z Generating XML reports...
2022-09-27T16:36:43.4826415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163643.xml
2022-09-27T16:36:43.9705581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoojiy2pn
2022-09-27T16:36:43.9706199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoojiy2pn/_remote_module_non_scriptable.py
2022-09-27T16:36:44.0895539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4t785h6c
2022-09-27T16:36:44.0896397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4t785h6c/_remote_module_non_scriptable.py
2022-09-27T16:36:44.1156800Z [1664296604.115046] [28fd6574525b:98972:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:36:44.1157338Z [1664296604.115046] [28fd6574525b:98972:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:36:44.1184738Z [1664296604.118081] [28fd6574525b:98973:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:36:44.1185250Z [1664296604.118081] [28fd6574525b:98973:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:36:44.4264440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg78qq_6x
2022-09-27T16:36:44.4265053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg78qq_6x/_remote_module_non_scriptable.py
2022-09-27T16:36:44.5127231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6_hcpxt
2022-09-27T16:36:44.5130592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6_hcpxt/_remote_module_non_scriptable.py
2022-09-27T16:36:44.5578305Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-09-27T16:36:44.5582096Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-09-27T16:36:44.5811076Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-09-27T16:36:44.5812024Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-09-27T16:36:45.0886150Z ok (6.353s)
2022-09-27T16:36:45.0886369Z 
2022-09-27T16:36:45.0886761Z ----------------------------------------------------------------------
2022-09-27T16:36:45.0887091Z Ran 1 test in 6.353s
2022-09-27T16:36:45.0887589Z 
2022-09-27T16:36:45.0887701Z OK
2022-09-27T16:36:45.0887846Z 
2022-09-27T16:36:45.0887988Z Generating XML reports...
2022-09-27T16:36:45.0938074Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163638.xml
2022-09-27T16:36:45.4971756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:45.4972276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:45.4973063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:45.4973559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:45.5194615Z ok (6.450s)
2022-09-27T16:36:45.5194948Z 
2022-09-27T16:36:45.5195503Z ----------------------------------------------------------------------
2022-09-27T16:36:45.5196065Z Ran 1 test in 6.450s
2022-09-27T16:36:45.5196371Z 
2022-09-27T16:36:45.5196536Z OK
2022-09-27T16:36:45.5196771Z 
2022-09-27T16:36:45.5196999Z Generating XML reports...
2022-09-27T16:36:45.5233220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163639.xml
2022-09-27T16:36:46.0068661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:46.0090090Z 
2022-09-27T16:36:46.0090404Z Running tests...
2022-09-27T16:36:46.0090851Z ----------------------------------------------------------------------
2022-09-27T16:36:46.0112397Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:46.0112937Z 
2022-09-27T16:36:46.0113207Z ----------------------------------------------------------------------
2022-09-27T16:36:46.0113564Z Ran 1 test in 0.002s
2022-09-27T16:36:46.0113734Z 
2022-09-27T16:36:46.0113849Z OK (skipped=1)
2022-09-27T16:36:46.0114009Z 
2022-09-27T16:36:46.0114943Z Generating XML reports...
2022-09-27T16:36:46.0149185Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163646.xml
2022-09-27T16:36:47.2762418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:47.2763390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:47.2764604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:47.2765594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:47.6458391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:47.6459383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:47.6460625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:47.6461613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:47.7684182Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:47.7708539Z 
2022-09-27T16:36:47.7709039Z Running tests...
2022-09-27T16:36:47.7709907Z ----------------------------------------------------------------------
2022-09-27T16:36:47.9188787Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:47.9189957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:47.9191719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:47.9192728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:48.1298018Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:36:48.1320078Z 
2022-09-27T16:36:48.1320947Z Running tests...
2022-09-27T16:36:48.1321793Z ----------------------------------------------------------------------
2022-09-27T16:36:48.4186194Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:48.4209092Z 
2022-09-27T16:36:48.4209646Z Running tests...
2022-09-27T16:36:48.4210508Z ----------------------------------------------------------------------
2022-09-27T16:36:48.4233143Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:48.4234333Z 
2022-09-27T16:36:48.4234924Z ----------------------------------------------------------------------
2022-09-27T16:36:48.4236159Z Ran 1 test in 0.002s
2022-09-27T16:36:48.4236454Z 
2022-09-27T16:36:48.4236653Z OK (skipped=1)
2022-09-27T16:36:48.4236942Z 
2022-09-27T16:36:48.4237172Z Generating XML reports...
2022-09-27T16:36:48.4270151Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163648.xml
2022-09-27T16:36:49.3884843Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:49.4177242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99363
2022-09-27T16:36:49.4186776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99364
2022-09-27T16:36:49.6987997Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:49.7183703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99366
2022-09-27T16:36:49.7190217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99367
2022-09-27T16:36:50.3590244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:50.3590788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:50.3592362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:50.3592867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:50.8987413Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:50.9009466Z 
2022-09-27T16:36:50.9009767Z Running tests...
2022-09-27T16:36:50.9010224Z ----------------------------------------------------------------------
2022-09-27T16:36:50.9030298Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:50.9030857Z 
2022-09-27T16:36:50.9031197Z ----------------------------------------------------------------------
2022-09-27T16:36:50.9031553Z Ran 1 test in 0.002s
2022-09-27T16:36:50.9031723Z 
2022-09-27T16:36:50.9031834Z OK (skipped=1)
2022-09-27T16:36:50.9031997Z 
2022-09-27T16:36:50.9032107Z Generating XML reports...
2022-09-27T16:36:50.9067424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163650.xml
2022-09-27T16:36:51.1089552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:51.1090065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:51.1090987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:51.1091779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:51.1306047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:51.1306570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:51.1308113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:51.1308609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:51.4424388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:51.4424884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:51.4426971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:51.4427485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:51.4487191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:51.4487935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:51.4491525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:51.4492015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:51.6214932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:51.6484503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:51.6485332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:51.6518853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:51.6526134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:51.7498963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:52.0429776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:36:52.0430333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:36:52.0431131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:52.0431834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:36:52.0539813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:36:52.1444440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:36:52.9358524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:52.9359557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:52.9361086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:52.9362060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:53.0876107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0kk6qwyv
2022-09-27T16:36:53.0877312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0kk6qwyv/_remote_module_non_scriptable.py
2022-09-27T16:36:53.1914379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrgafk0g
2022-09-27T16:36:53.1915820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrgafk0g/_remote_module_non_scriptable.py
2022-09-27T16:36:53.2191613Z [1664296613.218538] [28fd6574525b:99363:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:36:53.2192893Z [1664296613.218538] [28fd6574525b:99363:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:36:53.2263315Z [1664296613.225977] [28fd6574525b:99364:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:36:53.2263841Z [1664296613.225977] [28fd6574525b:99364:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:36:53.3699202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzx1uotzo
2022-09-27T16:36:53.3700394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzx1uotzo/_remote_module_non_scriptable.py
2022-09-27T16:36:53.4715827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo256w0pm
2022-09-27T16:36:53.4717069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo256w0pm/_remote_module_non_scriptable.py
2022-09-27T16:36:53.4750764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:53.4775705Z 
2022-09-27T16:36:53.4776054Z Running tests...
2022-09-27T16:36:53.4776903Z ----------------------------------------------------------------------
2022-09-27T16:36:53.4798350Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:53.4799381Z 
2022-09-27T16:36:53.4799934Z ----------------------------------------------------------------------
2022-09-27T16:36:53.4800886Z Ran 1 test in 0.002s
2022-09-27T16:36:53.4801208Z 
2022-09-27T16:36:53.4801392Z OK (skipped=1)
2022-09-27T16:36:53.4801731Z 
2022-09-27T16:36:53.4801978Z Generating XML reports...
2022-09-27T16:36:53.4834966Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163653.xml
2022-09-27T16:36:53.6125191Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:36:53.6126590Z   warnings.warn(
2022-09-27T16:36:53.6128526Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:36:53.6129893Z   warnings.warn(
2022-09-27T16:36:53.9490769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:53.9491765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:53.9722467Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpzx1uotzo'>
2022-09-27T16:36:53.9723463Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:36:53.9724683Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpo256w0pm'>
2022-09-27T16:36:53.9725611Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:36:54.0442248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:54.0443170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:54.0500586Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:36:54.0502215Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:36:54.0612100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:54.0613328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:54.0677859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:54.0678828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:36:54.5314159Z ok (6.398s)
2022-09-27T16:36:54.5315004Z 
2022-09-27T16:36:54.5315614Z ----------------------------------------------------------------------
2022-09-27T16:36:54.5316441Z Ran 1 test in 6.399s
2022-09-27T16:36:54.5316722Z 
2022-09-27T16:36:54.5316881Z OK
2022-09-27T16:36:54.5317113Z 
2022-09-27T16:36:54.5317337Z Generating XML reports...
2022-09-27T16:36:54.5382443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163648.xml
2022-09-27T16:36:54.6299249Z ok (6.859s)
2022-09-27T16:36:54.6299474Z 
2022-09-27T16:36:54.6299880Z ----------------------------------------------------------------------
2022-09-27T16:36:54.6300222Z Ran 1 test in 6.859s
2022-09-27T16:36:54.6300391Z 
2022-09-27T16:36:54.6300467Z OK
2022-09-27T16:36:54.6300609Z 
2022-09-27T16:36:54.6300751Z Generating XML reports...
2022-09-27T16:36:54.6339250Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163647.xml
2022-09-27T16:36:55.5361662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:55.5362184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:55.5364227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:55.5364747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:56.1303666Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:56.1326754Z 
2022-09-27T16:36:56.1327026Z Running tests...
2022-09-27T16:36:56.1327468Z ----------------------------------------------------------------------
2022-09-27T16:36:56.1348796Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:56.1349381Z 
2022-09-27T16:36:56.1349667Z ----------------------------------------------------------------------
2022-09-27T16:36:56.1349985Z Ran 1 test in 0.002s
2022-09-27T16:36:56.1350150Z 
2022-09-27T16:36:56.1350260Z OK (skipped=1)
2022-09-27T16:36:56.1350415Z 
2022-09-27T16:36:56.1350545Z Generating XML reports...
2022-09-27T16:36:56.1385219Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163656.xml
2022-09-27T16:36:56.6674231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:56.6674771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:56.6675757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:56.6676248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:56.7635967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:56.7636487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:56.7638368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:56.7638886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:57.1511119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:36:57.1535039Z 
2022-09-27T16:36:57.1535774Z Running tests...
2022-09-27T16:36:57.1536702Z ----------------------------------------------------------------------
2022-09-27T16:36:57.2442628Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:57.2466674Z 
2022-09-27T16:36:57.2466841Z Running tests...
2022-09-27T16:36:57.2467554Z ----------------------------------------------------------------------
2022-09-27T16:36:57.2494836Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-09-27T16:36:57.2495221Z 
2022-09-27T16:36:57.2495785Z ----------------------------------------------------------------------
2022-09-27T16:36:57.2496500Z Ran 1 test in 0.003s
2022-09-27T16:36:57.2496836Z 
2022-09-27T16:36:57.2497050Z OK (skipped=1)
2022-09-27T16:36:57.2497288Z 
2022-09-27T16:36:57.2497429Z Generating XML reports...
2022-09-27T16:36:57.2531596Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163657.xml
2022-09-27T16:36:58.0596171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:58.0596987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:58.0598261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:58.0598744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:58.6198019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:36:58.6221017Z 
2022-09-27T16:36:58.6221480Z Running tests...
2022-09-27T16:36:58.6221976Z ----------------------------------------------------------------------
2022-09-27T16:36:58.6242600Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:36:58.6243461Z 
2022-09-27T16:36:58.6243800Z ----------------------------------------------------------------------
2022-09-27T16:36:58.6244154Z Ran 1 test in 0.002s
2022-09-27T16:36:58.6244320Z 
2022-09-27T16:36:58.6244415Z OK (skipped=1)
2022-09-27T16:36:58.6244578Z 
2022-09-27T16:36:58.6244704Z Generating XML reports...
2022-09-27T16:36:58.6278645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163658.xml
2022-09-27T16:36:58.7818164Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:36:58.8017381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99727
2022-09-27T16:36:58.8023396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99728
2022-09-27T16:36:59.1874737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:36:59.1875249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:36:59.1876322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:36:59.1876859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:36:59.7961508Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:36:59.7984776Z 
2022-09-27T16:36:59.7985257Z Running tests...
2022-09-27T16:36:59.7985745Z ----------------------------------------------------------------------
2022-09-27T16:37:00.4792636Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:00.4793628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:00.4794246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:00.4794734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:00.5258399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:00.5259071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:00.5260601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:00.5261089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:00.6130137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:00.6130621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:00.6133601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:00.6134093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:01.0633154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:01.0633710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:01.0634526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:01.0635242Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:01.0642023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:01.0642503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:01.1165700Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:01.1189304Z 
2022-09-27T16:37:01.1189878Z Running tests...
2022-09-27T16:37:01.1190477Z ----------------------------------------------------------------------
2022-09-27T16:37:01.1211316Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:37:01.1212095Z 
2022-09-27T16:37:01.1212412Z ----------------------------------------------------------------------
2022-09-27T16:37:01.1212757Z Ran 1 test in 0.002s
2022-09-27T16:37:01.1212924Z 
2022-09-27T16:37:01.1213018Z OK (skipped=1)
2022-09-27T16:37:01.1213478Z 
2022-09-27T16:37:01.1213627Z Generating XML reports...
2022-09-27T16:37:01.1247648Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163701.xml
2022-09-27T16:37:01.3983820Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:01.4188531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99839
2022-09-27T16:37:01.4193631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99840
2022-09-27T16:37:02.4927620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6pgj1gm
2022-09-27T16:37:02.4929377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6pgj1gm/_remote_module_non_scriptable.py
2022-09-27T16:37:02.5290630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8vyy1dx
2022-09-27T16:37:02.5291720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8vyy1dx/_remote_module_non_scriptable.py
2022-09-27T16:37:02.9953188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:02.9953762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:02.9971516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:02.9972317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:03.1230152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:03.1230708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:03.1231914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:03.1232392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:03.1516374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:03.1516889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:03.1520743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:03.1521699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:03.2462802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:03.2463726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:03.2464944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:03.2465839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:03.6643906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:03.6658245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:03.6659780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:03.6744528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:03.6750971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:03.7549728Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:03.7572920Z 
2022-09-27T16:37:03.7573215Z Running tests...
2022-09-27T16:37:03.7574005Z ----------------------------------------------------------------------
2022-09-27T16:37:03.7594394Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-09-27T16:37:03.7594861Z 
2022-09-27T16:37:03.7595162Z ----------------------------------------------------------------------
2022-09-27T16:37:03.7595492Z Ran 1 test in 0.002s
2022-09-27T16:37:03.7595664Z 
2022-09-27T16:37:03.7595776Z OK (skipped=1)
2022-09-27T16:37:03.7595934Z 
2022-09-27T16:37:03.7596061Z Generating XML reports...
2022-09-27T16:37:03.7630596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163703.xml
2022-09-27T16:37:03.7669876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:03.8139556Z ok (6.660s)
2022-09-27T16:37:03.8139779Z 
2022-09-27T16:37:03.8140157Z ----------------------------------------------------------------------
2022-09-27T16:37:03.8140504Z Ran 1 test in 6.660s
2022-09-27T16:37:03.8140658Z 
2022-09-27T16:37:03.8140765Z OK
2022-09-27T16:37:03.8140906Z 
2022-09-27T16:37:03.8141065Z Generating XML reports...
2022-09-27T16:37:03.8183795Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163657.xml
2022-09-27T16:37:05.1499617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjgi0rf4
2022-09-27T16:37:05.1500829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjgi0rf4/_remote_module_non_scriptable.py
2022-09-27T16:37:05.2245216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzhgw26u
2022-09-27T16:37:05.2246100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzhgw26u/_remote_module_non_scriptable.py
2022-09-27T16:37:05.2508860Z [1664296625.250439] [28fd6574525b:99839:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:05.2509370Z [1664296625.250439] [28fd6574525b:99839:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:05.2572331Z [1664296625.256833] [28fd6574525b:99840:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:05.2572867Z [1664296625.256833] [28fd6574525b:99840:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:05.8413663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:05.8414235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:05.8415072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:05.8415563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:05.9904374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:05.9904913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:05.9907239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:05.9907747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:05.9998656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:05.9999179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:06.0086089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:06.0086594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:06.3659525Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:06.3683175Z 
2022-09-27T16:37:06.3683712Z Running tests...
2022-09-27T16:37:06.3684546Z ----------------------------------------------------------------------
2022-09-27T16:37:06.4980302Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:37:06.5002084Z 
2022-09-27T16:37:06.5002284Z Running tests...
2022-09-27T16:37:06.5002729Z ----------------------------------------------------------------------
2022-09-27T16:37:06.6316927Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... ok (6.833s)
2022-09-27T16:37:06.6317275Z 
2022-09-27T16:37:06.6317761Z ----------------------------------------------------------------------
2022-09-27T16:37:06.6318092Z Ran 1 test in 6.833s
2022-09-27T16:37:06.6318266Z 
2022-09-27T16:37:06.6318369Z OK
2022-09-27T16:37:06.6318506Z 
2022-09-27T16:37:06.6318634Z Generating XML reports...
2022-09-27T16:37:06.6350142Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163659.xml
2022-09-27T16:37:08.0500134Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:08.0700135Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100060
2022-09-27T16:37:08.0706327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100061
2022-09-27T16:37:08.2109964Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:08.2302530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100063
2022-09-27T16:37:08.2308362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100064
2022-09-27T16:37:08.8039442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:08.8042009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:08.8049238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:08.8050234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:09.4150075Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:37:09.4172815Z 
2022-09-27T16:37:09.4172986Z Running tests...
2022-09-27T16:37:09.4173461Z ----------------------------------------------------------------------
2022-09-27T16:37:09.8320593Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:09.8321227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:09.8323036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:09.8323523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:09.8448824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:09.8449327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:09.8452465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:09.8452946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:09.8513635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:09.8514114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:09.8518106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:09.8518577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:10.0026251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:10.0027079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:10.0028159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:10.0028654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:10.3671933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:10.3687479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:10.3688275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:10.3773025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:10.3779513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:10.4703107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:10.5486867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:10.5487716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:10.5488503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:10.5489210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:10.5598228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:10.6502737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:10.6712705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:37:10.6713853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:37:10.6715326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:37:10.6716753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:37:11.0828017Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:11.1038765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100220
2022-09-27T16:37:11.1045610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100221
2022-09-27T16:37:11.7842886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7aga6ku6
2022-09-27T16:37:11.7845024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7aga6ku6/_remote_module_non_scriptable.py
2022-09-27T16:37:11.9145806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyh3cs9xh
2022-09-27T16:37:11.9148233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyh3cs9xh/_remote_module_non_scriptable.py
2022-09-27T16:37:12.1401752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2ab3acv
2022-09-27T16:37:12.1402380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2ab3acv/_remote_module_non_scriptable.py
2022-09-27T16:37:12.1422094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5n3q45px
2022-09-27T16:37:12.1425024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5n3q45px/_remote_module_non_scriptable.py
2022-09-27T16:37:12.6210522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:12.6211072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:12.6227570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:12.6228325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:12.7979128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:12.7979649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:12.7981129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:12.7981614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:12.7998343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:12.7998797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:12.8003554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:12.8004060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:13.3426742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:13.3521358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:13.3522885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:13.3529172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:13.3536609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:13.4536452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:13.4588226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.4588773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.4728728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.4729246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.4848958Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:37:13.4850566Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:37:13.5183158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.5183687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.5321812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.5322318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.5417469Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:37:13.5418274Z   warnings.warn(
2022-09-27T16:37:13.5419354Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:37:13.5420060Z   warnings.warn(
2022-09-27T16:37:13.5432977Z ok (7.043s)
2022-09-27T16:37:13.5433138Z 
2022-09-27T16:37:13.5433704Z ----------------------------------------------------------------------
2022-09-27T16:37:13.5434100Z Ran 1 test in 7.043s
2022-09-27T16:37:13.5434281Z 
2022-09-27T16:37:13.5434377Z OK
2022-09-27T16:37:13.5434516Z 
2022-09-27T16:37:13.5434649Z Generating XML reports...
2022-09-27T16:37:13.5468647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.5469386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.5470542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163706.xml
2022-09-27T16:37:13.5614008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:13.5614514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:14.2841479Z ok (7.915s)
2022-09-27T16:37:14.2841712Z 
2022-09-27T16:37:14.2842117Z ----------------------------------------------------------------------
2022-09-27T16:37:14.2842463Z Ran 1 test in 7.916s
2022-09-27T16:37:14.2842629Z 
2022-09-27T16:37:14.2842748Z OK
2022-09-27T16:37:14.2842888Z 
2022-09-27T16:37:14.2843007Z Generating XML reports...
2022-09-27T16:37:14.2879314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163706.xml
2022-09-27T16:37:14.9659184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx3ox81d
2022-09-27T16:37:14.9659792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx3ox81d/_remote_module_non_scriptable.py
2022-09-27T16:37:14.9829899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctoanx65
2022-09-27T16:37:14.9830878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctoanx65/_remote_module_non_scriptable.py
2022-09-27T16:37:15.0227541Z [1664296635.022257] [28fd6574525b:100221:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:15.0228061Z [1664296635.022257] [28fd6574525b:100221:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:15.0319413Z [1664296635.031576] [28fd6574525b:100220:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:15.0319923Z [1664296635.031576] [28fd6574525b:100220:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:15.6915303Z STAGE:2022-09-27 16:37:15 100221:100221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:37:15.7034242Z STAGE:2022-09-27 16:37:15 100220:100220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:37:15.7112364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:15.7113390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:15.7294827Z STAGE:2022-09-27 16:37:15 100221:100221 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:37:15.7296431Z STAGE:2022-09-27 16:37:15 100220:100220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:37:15.7928422Z STAGE:2022-09-27 16:37:15 100220:100220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:37:15.9111799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:15.9112730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:15.9113798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:15.9114638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:16.4171683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:37:16.4194498Z 
2022-09-27T16:37:16.4194947Z Running tests...
2022-09-27T16:37:16.4195459Z ----------------------------------------------------------------------
2022-09-27T16:37:16.4475315Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:16.4476570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:16.4478776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:16.4479275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:16.6519677Z STAGE:2022-09-27 16:37:16 100220:100220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:37:16.9540237Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:16.9562981Z 
2022-09-27T16:37:16.9563153Z Running tests...
2022-09-27T16:37:16.9564279Z ----------------------------------------------------------------------
2022-09-27T16:37:17.4191291Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... ok (8.001s)
2022-09-27T16:37:17.4191877Z 
2022-09-27T16:37:17.4192559Z ----------------------------------------------------------------------
2022-09-27T16:37:17.4193230Z Ran 1 test in 8.002s
2022-09-27T16:37:17.4193559Z 
2022-09-27T16:37:17.4193738Z OK
2022-09-27T16:37:17.4193974Z 
2022-09-27T16:37:17.4194225Z Generating XML reports...
2022-09-27T16:37:17.4227947Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163709.xml
2022-09-27T16:37:18.0949232Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:18.1152414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100393
2022-09-27T16:37:18.1158455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100394
2022-09-27T16:37:18.7127311Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:18.7324926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100427
2022-09-27T16:37:18.7331469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100428
2022-09-27T16:37:19.6454772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:19.6455308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:19.6456281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:19.6456756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:19.8572392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:19.8572916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:19.8574809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:19.8575591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:19.8652528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:19.8653016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:19.8656515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:19.8657002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:20.1512067Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:37:20.1535648Z 
2022-09-27T16:37:20.1535804Z Running tests...
2022-09-27T16:37:20.1536534Z ----------------------------------------------------------------------
2022-09-27T16:37:20.3911628Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:20.3912270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:20.3913071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:20.3914028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:20.4019723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:20.4022039Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:37:20.4332010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:20.4332509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:20.4335426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:20.4335897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:20.4395693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:20.4396161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:20.4400033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:20.4400743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:20.4926204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:20.4926804Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:37:20.9816814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:20.9902232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:20.9903059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:20.9917658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:20.9924782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:21.0918448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:21.7856440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehsp24gv
2022-09-27T16:37:21.7857074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehsp24gv/_remote_module_non_scriptable.py
2022-09-27T16:37:21.8372471Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:21.8534069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rs08405
2022-09-27T16:37:21.8535235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rs08405/_remote_module_non_scriptable.py
2022-09-27T16:37:21.8578623Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100578
2022-09-27T16:37:21.8585249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100579
2022-09-27T16:37:22.3416529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfltlkeco
2022-09-27T16:37:22.3417134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfltlkeco/_remote_module_non_scriptable.py
2022-09-27T16:37:22.3738382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.3738928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.3756970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.3757703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.4021715Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:37:22.4022323Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:37:22.4163545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjf8an15
2022-09-27T16:37:22.4164627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjf8an15/_remote_module_non_scriptable.py
2022-09-27T16:37:22.7847359Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:37:22.7847982Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-09-27T16:37:22.7929470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.7929975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.7948114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.7948608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.8207052Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:37:22.8207667Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-09-27T16:37:22.9909559Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:37:22.9910161Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-09-27T16:37:22.9991673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:22.9992183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:23.0010808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:23.0011302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:23.5227777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:23.5228574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:23.5229841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:23.5230324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:23.5717280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:23.5717943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:23.5719076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:23.5719554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:23.9288418Z ok (7.509s)
2022-09-27T16:37:23.9288647Z 
2022-09-27T16:37:23.9289048Z ----------------------------------------------------------------------
2022-09-27T16:37:23.9289388Z Ran 1 test in 7.509s
2022-09-27T16:37:23.9289557Z 
2022-09-27T16:37:23.9289636Z OK
2022-09-27T16:37:23.9289771Z 
2022-09-27T16:37:23.9289911Z Generating XML reports...
2022-09-27T16:37:23.9327552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163716.xml
2022-09-27T16:37:23.9444166Z ok (6.988s)
2022-09-27T16:37:23.9444338Z 
2022-09-27T16:37:23.9444651Z ----------------------------------------------------------------------
2022-09-27T16:37:23.9444995Z Ran 1 test in 6.988s
2022-09-27T16:37:23.9445160Z 
2022-09-27T16:37:23.9445517Z OK
2022-09-27T16:37:23.9445660Z 
2022-09-27T16:37:23.9445788Z Generating XML reports...
2022-09-27T16:37:23.9482082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163716.xml
2022-09-27T16:37:24.0759920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:24.0973475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:24.0974273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:24.1065663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:24.1073018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:24.1988852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:25.6139580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7o_00wj
2022-09-27T16:37:25.6212412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7o_00wj/_remote_module_non_scriptable.py
2022-09-27T16:37:25.6212965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1oye7tpu
2022-09-27T16:37:25.6215160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1oye7tpu/_remote_module_non_scriptable.py
2022-09-27T16:37:25.6496682Z [1664296645.649298] [28fd6574525b:100578:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:25.6497204Z [1664296645.649298] [28fd6574525b:100578:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:25.6574845Z [1664296645.657163] [28fd6574525b:100579:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:25.6575336Z [1664296645.657163] [28fd6574525b:100579:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:26.1869993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:26.1870514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:26.1871863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:26.1872347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:26.2517774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:26.2518557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:26.2519957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:26.2520919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:26.5690738Z ok (6.415s)
2022-09-27T16:37:26.5690953Z 
2022-09-27T16:37:26.5691344Z ----------------------------------------------------------------------
2022-09-27T16:37:26.5691684Z Ran 1 test in 6.415s
2022-09-27T16:37:26.5691852Z 
2022-09-27T16:37:26.5691946Z OK
2022-09-27T16:37:26.5692082Z 
2022-09-27T16:37:26.5692221Z Generating XML reports...
2022-09-27T16:37:26.5733466Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163720.xml
2022-09-27T16:37:26.7045749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:37:26.7068704Z 
2022-09-27T16:37:26.7068847Z Running tests...
2022-09-27T16:37:26.7069662Z ----------------------------------------------------------------------
2022-09-27T16:37:26.7615915Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:26.7639515Z 
2022-09-27T16:37:26.7639823Z Running tests...
2022-09-27T16:37:26.7640509Z ----------------------------------------------------------------------
2022-09-27T16:37:28.3367987Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:28.3568951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100769
2022-09-27T16:37:28.3574751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100770
2022-09-27T16:37:28.3629604Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:28.4178633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100772
2022-09-27T16:37:28.4183811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100773
2022-09-27T16:37:28.6553021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:28.6553538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:28.6555655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:28.6556148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:29.3833398Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:37:29.3856086Z 
2022-09-27T16:37:29.3856319Z Running tests...
2022-09-27T16:37:29.3856773Z ----------------------------------------------------------------------
2022-09-27T16:37:30.1141648Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:30.1142277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:30.1143675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:30.1144177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:30.1146559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:30.1147031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:30.1150890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:30.1151381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:30.1711536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:30.1712055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:30.1713437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:30.1713936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:30.1991436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:30.1991938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:30.1994037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:30.1994536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:30.6392992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:30.6456552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:30.6457347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:30.6494466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:30.6500772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:30.7435233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:30.7435803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:30.7436588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:30.7437315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:30.7469400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:30.7544590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:30.7547023Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:37:30.8449957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:30.8451243Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:37:31.0054294Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:31.0268400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100923
2022-09-27T16:37:31.0272758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100924
2022-09-27T16:37:32.1982326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyf3h8tbo
2022-09-27T16:37:32.1982943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyf3h8tbo/_remote_module_non_scriptable.py
2022-09-27T16:37:32.2476808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjull5nw0
2022-09-27T16:37:32.2477439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjull5nw0/_remote_module_non_scriptable.py
2022-09-27T16:37:32.5332197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkhnl6dx
2022-09-27T16:37:32.5332864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkhnl6dx/_remote_module_non_scriptable.py
2022-09-27T16:37:32.6175399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphefre8od
2022-09-27T16:37:32.6176039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphefre8od/_remote_module_non_scriptable.py
2022-09-27T16:37:32.7134370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:32.7134934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:32.7152633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:32.7153157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:32.7159966Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:37:32.7161432Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:37:32.7194839Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:37:32.7195985Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:37:32.7197722Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:37:32.7199019Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:37:32.7200524Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:37:32.7201269Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:37:32.7302763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:32.7303279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:32.7307767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:32.7308667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:32.7435313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:32.7435826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:32.7438435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:32.7438935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:33.0621102Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:37:33.0622306Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:37:33.0707389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:33.0708209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:33.0725380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:33.0725893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:33.0732073Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:37:33.0733118Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations.
2022-09-27T16:37:33.0766186Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:37:33.0766818Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:37:33.0767970Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:37:33.0768657Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:37:33.2653541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:33.2822838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:33.2823876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:33.2857076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:33.2864092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:33.3838151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:33.8051086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:33.8051645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:33.8133770Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpvkhnl6dx'>
2022-09-27T16:37:33.8134301Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:37:33.8134908Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmphefre8od'>
2022-09-27T16:37:33.8135368Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:37:34.0318260Z ok (7.325s)
2022-09-27T16:37:34.0318533Z 
2022-09-27T16:37:34.0318944Z ----------------------------------------------------------------------
2022-09-27T16:37:34.0319294Z Ran 1 test in 7.325s
2022-09-27T16:37:34.0319463Z 
2022-09-27T16:37:34.0319559Z OK
2022-09-27T16:37:34.0319682Z 
2022-09-27T16:37:34.0319842Z Generating XML reports...
2022-09-27T16:37:34.0356839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163726.xml
2022-09-27T16:37:34.4708612Z ok (7.707s)
2022-09-27T16:37:34.4708846Z 
2022-09-27T16:37:34.4709226Z ----------------------------------------------------------------------
2022-09-27T16:37:34.4709552Z Ran 1 test in 7.707s
2022-09-27T16:37:34.4709724Z 
2022-09-27T16:37:34.4709827Z OK
2022-09-27T16:37:34.4709961Z 
2022-09-27T16:37:34.4710096Z Generating XML reports...
2022-09-27T16:37:34.4747063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163726.xml
2022-09-27T16:37:34.8704964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0a1437zy
2022-09-27T16:37:34.8706225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0a1437zy/_remote_module_non_scriptable.py
2022-09-27T16:37:34.8825741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogjy55sz
2022-09-27T16:37:34.8827778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogjy55sz/_remote_module_non_scriptable.py
2022-09-27T16:37:34.9181755Z [1664296654.917771] [28fd6574525b:100923:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:34.9182275Z [1664296654.917771] [28fd6574525b:100923:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:34.9200676Z [1664296654.919687] [28fd6574525b:100924:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:34.9201287Z [1664296654.919687] [28fd6574525b:100924:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:35.8383006Z ok (6.452s)
2022-09-27T16:37:35.8383284Z 
2022-09-27T16:37:35.8383687Z ----------------------------------------------------------------------
2022-09-27T16:37:35.8384052Z Ran 1 test in 6.453s
2022-09-27T16:37:35.8384225Z 
2022-09-27T16:37:35.8384321Z OK
2022-09-27T16:37:35.8384460Z 
2022-09-27T16:37:35.8384584Z Generating XML reports...
2022-09-27T16:37:35.8425554Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163729.xml
2022-09-27T16:37:36.3328196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:36.3328694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:36.3330992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:36.3331490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:36.6565967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:36.6566469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:36.6567918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:36.6568415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:36.8311004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:37:36.8335057Z 
2022-09-27T16:37:36.8335319Z Running tests...
2022-09-27T16:37:36.8335766Z ----------------------------------------------------------------------
2022-09-27T16:37:37.1467836Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:37.1489487Z 
2022-09-27T16:37:37.1489802Z Running tests...
2022-09-27T16:37:37.1490245Z ----------------------------------------------------------------------
2022-09-27T16:37:37.1546244Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-09-27T16:37:37.1546587Z 
2022-09-27T16:37:37.1546875Z ----------------------------------------------------------------------
2022-09-27T16:37:37.1547229Z Ran 1 test in 0.006s
2022-09-27T16:37:37.1547377Z 
2022-09-27T16:37:37.1547490Z OK (skipped=1)
2022-09-27T16:37:37.1547648Z 
2022-09-27T16:37:37.1547775Z Generating XML reports...
2022-09-27T16:37:37.1582026Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163737.xml
2022-09-27T16:37:37.8715796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:37.8716315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:37.8717439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:37.8718286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:38.3652372Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:37:38.3675001Z 
2022-09-27T16:37:38.3675186Z Running tests...
2022-09-27T16:37:38.3675982Z ----------------------------------------------------------------------
2022-09-27T16:37:38.4584373Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:38.4787148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101152
2022-09-27T16:37:38.4794379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101153
2022-09-27T16:37:39.0738096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:39.0738620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:39.0740144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:39.0740645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:39.6604181Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:39.6627153Z 
2022-09-27T16:37:39.6627414Z Running tests...
2022-09-27T16:37:39.6627867Z ----------------------------------------------------------------------
2022-09-27T16:37:39.9901281Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:40.0104272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101219
2022-09-27T16:37:40.0110010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101220
2022-09-27T16:37:40.1229007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:40.1229507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:40.1230752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:40.1231250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:40.1290097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:40.1290545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:40.1293801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:40.1294295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:40.6688267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:40.6692001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:40.6696487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:40.6791068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:40.6809885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:40.6812250Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:37:40.6889180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:40.6896671Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-09-27T16:37:41.3743310Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:41.3950893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101296
2022-09-27T16:37:41.3956108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101297
2022-09-27T16:37:41.6501900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:41.6502431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:41.6504061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:41.6504937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:41.7350160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:41.7350672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:41.7352336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:41.7352828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:42.0556639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0lr99xj
2022-09-27T16:37:42.0558062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0lr99xj/_remote_module_non_scriptable.py
2022-09-27T16:37:42.0666721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh2ujgui
2022-09-27T16:37:42.0672486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh2ujgui/_remote_module_non_scriptable.py
2022-09-27T16:37:42.2613274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:42.3694655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:42.3695535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:42.3729742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:42.3736879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:42.4708052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:42.5820123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:42.5820675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:42.5856619Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-09-27T16:37:42.5857223Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-09-27T16:37:42.5871216Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:37:42.5871963Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-09-27T16:37:42.5873187Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:37:42.5873998Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-09-27T16:37:42.5876129Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:37:42.5877705Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-09-27T16:37:42.6167965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:42.6168522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:43.1412961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:43.1413481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:43.1414069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:43.1414565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:43.1649077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:43.1649566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:43.1652000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:43.1652497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:43.2907314Z ok (6.457s)
2022-09-27T16:37:43.2907670Z 
2022-09-27T16:37:43.2908449Z ----------------------------------------------------------------------
2022-09-27T16:37:43.2909022Z Ran 1 test in 6.457s
2022-09-27T16:37:43.2909177Z 
2022-09-27T16:37:43.2909270Z OK
2022-09-27T16:37:43.2909405Z 
2022-09-27T16:37:43.2909539Z Generating XML reports...
2022-09-27T16:37:43.2946067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163736.xml
2022-09-27T16:37:43.7335704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:43.7497867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:43.7499134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:43.7538959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:43.7545473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:43.7566633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsw52gt_d
2022-09-27T16:37:43.7569501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsw52gt_d/_remote_module_non_scriptable.py
2022-09-27T16:37:43.8511086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:43.8945125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmput1yak0r
2022-09-27T16:37:43.8946280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmput1yak0r/_remote_module_non_scriptable.py
2022-09-27T16:37:43.9256039Z [1664296663.925056] [28fd6574525b:101220:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:43.9256905Z [1664296663.925056] [28fd6574525b:101220:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:43.9263263Z [1664296663.925987] [28fd6574525b:101219:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:37:43.9264302Z [1664296663.925987] [28fd6574525b:101219:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:37:44.7862760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:44.7863632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:45.1947152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfd_0k_g0
2022-09-27T16:37:45.1948483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfd_0k_g0/_remote_module_non_scriptable.py
2022-09-27T16:37:45.3087041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5_q65lb
2022-09-27T16:37:45.3087995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5_q65lb/_remote_module_non_scriptable.py
2022-09-27T16:37:45.4233571Z ok (7.055s)
2022-09-27T16:37:45.4233788Z 
2022-09-27T16:37:45.4234185Z ----------------------------------------------------------------------
2022-09-27T16:37:45.4234512Z Ran 1 test in 7.056s
2022-09-27T16:37:45.4234725Z 
2022-09-27T16:37:45.4234820Z OK
2022-09-27T16:37:45.4234958Z 
2022-09-27T16:37:45.4235098Z Generating XML reports...
2022-09-27T16:37:45.4271808Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163738.xml
2022-09-27T16:37:45.5361399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:45.5362211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:45.5363285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:45.5363756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:46.0633109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:37:46.0655877Z 
2022-09-27T16:37:46.0656176Z Running tests...
2022-09-27T16:37:46.0656628Z ----------------------------------------------------------------------
2022-09-27T16:37:46.9603737Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:46.9604419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:47.5989115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:47.5989634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:47.5991422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:47.5991918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:47.6091818Z ok (7.946s)
2022-09-27T16:37:47.6092012Z 
2022-09-27T16:37:47.6092550Z ----------------------------------------------------------------------
2022-09-27T16:37:47.6092913Z Ran 1 test in 7.946s
2022-09-27T16:37:47.6093078Z 
2022-09-27T16:37:47.6093170Z OK
2022-09-27T16:37:47.6093304Z 
2022-09-27T16:37:47.6093457Z Generating XML reports...
2022-09-27T16:37:47.6129544Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163739.xml
2022-09-27T16:37:47.7581714Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:47.7784278Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101473
2022-09-27T16:37:47.7790069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101474
2022-09-27T16:37:48.1180395Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:37:48.1203445Z 
2022-09-27T16:37:48.1203842Z Running tests...
2022-09-27T16:37:48.1204369Z ----------------------------------------------------------------------
2022-09-27T16:37:49.4519370Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:49.4520553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:49.4521216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:49.4521694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:49.4643783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:49.4644272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:49.4647065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:49.4647542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:49.7781529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:49.7782069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:49.7784315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:49.7784793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:49.8098231Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:49.8267263Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.706s)
2022-09-27T16:37:49.8267922Z 
2022-09-27T16:37:49.8268201Z ----------------------------------------------------------------------
2022-09-27T16:37:49.8268533Z Ran 1 test in 1.706s
2022-09-27T16:37:49.8268700Z 
2022-09-27T16:37:49.8268793Z OK (skipped=1)
2022-09-27T16:37:49.8268948Z 
2022-09-27T16:37:49.8269079Z Generating XML reports...
2022-09-27T16:37:49.8303131Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163748.xml
2022-09-27T16:37:49.9994332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:49.9994891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:49.9995663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:49.9996362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:50.0002501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:50.0003522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:50.2858246Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:50.2880931Z 
2022-09-27T16:37:50.2881198Z Running tests...
2022-09-27T16:37:50.2881625Z ----------------------------------------------------------------------
2022-09-27T16:37:51.4629691Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolhmh_7c
2022-09-27T16:37:51.4634097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolhmh_7c/_remote_module_non_scriptable.py
2022-09-27T16:37:51.5041498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_i76454
2022-09-27T16:37:51.5042733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_i76454/_remote_module_non_scriptable.py
2022-09-27T16:37:51.9145896Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:51.9349948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101647
2022-09-27T16:37:51.9355633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101648
2022-09-27T16:37:51.9620452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:51.9620985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:51.9622283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:51.9622773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:51.9982133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:51.9982689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:52.0826746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:52.0827311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:37:52.4943539Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:37:52.4966859Z 
2022-09-27T16:37:52.4967151Z Running tests...
2022-09-27T16:37:52.4967893Z ----------------------------------------------------------------------
2022-09-27T16:37:52.7904970Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... ok (6.724s)
2022-09-27T16:37:52.7905602Z 
2022-09-27T16:37:52.7906277Z ----------------------------------------------------------------------
2022-09-27T16:37:52.7906929Z Ran 1 test in 6.725s
2022-09-27T16:37:52.7907242Z 
2022-09-27T16:37:52.7907416Z OK
2022-09-27T16:37:52.7907672Z 
2022-09-27T16:37:52.7907884Z Generating XML reports...
2022-09-27T16:37:52.7944486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163746.xml
2022-09-27T16:37:53.6038842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:53.6039371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:53.6039941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:53.6040696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:53.6041304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:53.6041791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:53.6042685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:53.6043173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:54.1288458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:54.1296410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:54.1297212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:54.1389254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:54.1395645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:54.2309975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:54.2520174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:37:54.2520936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:37:54.2521926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:37:54.2522650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:37:54.2524406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:37:54.2524909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:37:54.2525589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:37:54.2526265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:37:54.2730214Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:54.2899615Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.793s)
2022-09-27T16:37:54.2900238Z 
2022-09-27T16:37:54.2900535Z ----------------------------------------------------------------------
2022-09-27T16:37:54.2901218Z Ran 1 test in 1.793s
2022-09-27T16:37:54.2901385Z 
2022-09-27T16:37:54.2901497Z OK (skipped=1)
2022-09-27T16:37:54.2901655Z 
2022-09-27T16:37:54.2901784Z Generating XML reports...
2022-09-27T16:37:54.2934172Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163752.xml
2022-09-27T16:37:54.9213568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:54.9214509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:54.9215649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:54.9216617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:55.4490964Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:37:55.4520346Z 
2022-09-27T16:37:55.4521309Z Running tests...
2022-09-27T16:37:55.4522347Z ----------------------------------------------------------------------
2022-09-27T16:37:55.6341998Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbvzcro2l
2022-09-27T16:37:55.6343432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbvzcro2l/_remote_module_non_scriptable.py
2022-09-27T16:37:55.6576030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumgb8wwg
2022-09-27T16:37:55.6577162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumgb8wwg/_remote_module_non_scriptable.py
2022-09-27T16:37:56.3475886Z ok (6.059s)
2022-09-27T16:37:56.3476139Z 
2022-09-27T16:37:56.3476521Z ----------------------------------------------------------------------
2022-09-27T16:37:56.3476871Z Ran 1 test in 6.059s
2022-09-27T16:37:56.3477055Z 
2022-09-27T16:37:56.3477150Z OK
2022-09-27T16:37:56.3477288Z 
2022-09-27T16:37:56.3478389Z Generating XML reports...
2022-09-27T16:37:56.3537254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163750.xml
2022-09-27T16:37:56.4486320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:56.4486953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:56.4488750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:56.4489519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:56.9615890Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:37:56.9636566Z 
2022-09-27T16:37:56.9637059Z Running tests...
2022-09-27T16:37:56.9637988Z ----------------------------------------------------------------------
2022-09-27T16:37:56.9667131Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T16:37:56.9667891Z 
2022-09-27T16:37:56.9668462Z ----------------------------------------------------------------------
2022-09-27T16:37:56.9668885Z Ran 1 test in 0.003s
2022-09-27T16:37:56.9669052Z 
2022-09-27T16:37:56.9669160Z OK (skipped=1)
2022-09-27T16:37:56.9669316Z 
2022-09-27T16:37:56.9669442Z Generating XML reports...
2022-09-27T16:37:56.9702830Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163756.xml
2022-09-27T16:37:57.1404966Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:37:57.1610120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101811
2022-09-27T16:37:57.1616066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101812
2022-09-27T16:37:58.6518867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:58.6519546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:58.6521149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:58.6521645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:58.9395487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:58.9395999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:58.9397601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:58.9398094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:58.9558834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:58.9559326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:58.9563088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:58.9563580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:59.0467306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:37:59.0467806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:37:59.0470284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:37:59.0470784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:37:59.1366920Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:37:59.1389981Z 
2022-09-27T16:37:59.1390442Z Running tests...
2022-09-27T16:37:59.1390941Z ----------------------------------------------------------------------
2022-09-27T16:37:59.4846143Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:37:59.4846805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:37:59.4847593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:59.4848281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:37:59.4854504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:37:59.4855025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:37:59.5484573Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:37:59.5509383Z 
2022-09-27T16:37:59.5509669Z Running tests...
2022-09-27T16:37:59.5510121Z ----------------------------------------------------------------------
2022-09-27T16:38:00.8347277Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:00.8554348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101955
2022-09-27T16:38:00.8560166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101956
2022-09-27T16:38:00.9188593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprsy9tsmx
2022-09-27T16:38:00.9189372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprsy9tsmx/_remote_module_non_scriptable.py
2022-09-27T16:38:00.9386487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphdto7y1c
2022-09-27T16:38:00.9387608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphdto7y1c/_remote_module_non_scriptable.py
2022-09-27T16:38:01.2165941Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:01.2376290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101988
2022-09-27T16:38:01.2382145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101989
2022-09-27T16:38:01.4287816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:01.4288337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:01.5018412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:01.5018995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:02.2729605Z ok (6.821s)
2022-09-27T16:38:02.2729833Z 
2022-09-27T16:38:02.2730227Z ----------------------------------------------------------------------
2022-09-27T16:38:02.2730575Z Ran 1 test in 6.821s
2022-09-27T16:38:02.2730743Z 
2022-09-27T16:38:02.2730841Z OK
2022-09-27T16:38:02.2730981Z 
2022-09-27T16:38:02.2731117Z Generating XML reports...
2022-09-27T16:38:02.2767603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163755.xml
2022-09-27T16:38:02.6506267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:02.6506783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:02.6508714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:02.6509209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:02.6716407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:02.6716910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:02.6719368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:02.6719845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:03.0121063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:03.0121563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:03.0122175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:03.0122942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:03.0613766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:03.0614238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:03.0617108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:03.0617597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:03.1671161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:03.1738160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:03.1740601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:03.1772293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:03.1778876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:03.2752676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:03.2963229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:38:03.2963748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:38:03.2964522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:38:03.2965235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:38:03.2966970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:38:03.2967490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:38:03.2968173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:38:03.2968880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:38:03.5751615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:03.6475120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:03.6475938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:03.6562718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:03.6569679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:03.6572732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:38:03.7488503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:03.7491033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:38:03.7491839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:38:03.7589535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:38:04.2456182Z ok (4.694s)
2022-09-27T16:38:04.2456387Z 
2022-09-27T16:38:04.2456786Z ----------------------------------------------------------------------
2022-09-27T16:38:04.2457113Z Ran 1 test in 4.695s
2022-09-27T16:38:04.2457285Z 
2022-09-27T16:38:04.2457615Z OK
2022-09-27T16:38:04.2457769Z 
2022-09-27T16:38:04.2457908Z Generating XML reports...
2022-09-27T16:38:04.2492553Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163759.xml
2022-09-27T16:38:04.4884157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:04.4884667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:04.4887072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:04.4887583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:04.7141903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ep9b_uf
2022-09-27T16:38:04.7143360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ep9b_uf/_remote_module_non_scriptable.py
2022-09-27T16:38:04.7457405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekydsyjn
2022-09-27T16:38:04.7458516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekydsyjn/_remote_module_non_scriptable.py
2022-09-27T16:38:05.0106141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:38:05.0129478Z 
2022-09-27T16:38:05.0129986Z Running tests...
2022-09-27T16:38:05.0130477Z ----------------------------------------------------------------------
2022-09-27T16:38:06.3899093Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:06.3899800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:06.3900925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:06.3901400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:06.9015835Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:06.9039716Z 
2022-09-27T16:38:06.9039955Z Running tests...
2022-09-27T16:38:06.9040814Z ----------------------------------------------------------------------
2022-09-27T16:38:15.2835744Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... [W ProcessGroupNCCL.cpp:948] [Rank 0] Found key in store: NCCLABORTEDCOMM:209a3bac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-09-27T16:38:15.2836849Z [W ProcessGroupNCCL.cpp:948] [Rank 1] Found key in store: NCCLABORTEDCOMM:209a3bac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-09-27T16:38:15.8829390Z ok (16.744s)
2022-09-27T16:38:15.8829606Z 
2022-09-27T16:38:15.8830003Z ----------------------------------------------------------------------
2022-09-27T16:38:15.8830323Z Ran 1 test in 16.744s
2022-09-27T16:38:15.8830492Z 
2022-09-27T16:38:15.8830587Z OK
2022-09-27T16:38:15.8830727Z 
2022-09-27T16:38:15.8830861Z Generating XML reports...
2022-09-27T16:38:15.8866353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163759.xml
2022-09-27T16:38:16.7896869Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:16.8096357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102217
2022-09-27T16:38:16.8103037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102218
2022-09-27T16:38:16.8241626Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:16.8442616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102220
2022-09-27T16:38:16.8449076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102221
2022-09-27T16:38:18.1176040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:18.1176584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:18.1177851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:18.1178340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:18.4759033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:18.4759623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:18.4760422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:18.4761250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:18.4985571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:18.4986051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:18.4988729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:18.4989232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:18.5263604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:18.5264077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:18.5268094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:18.5268596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:18.5470993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:18.5471479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:18.5473736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:18.5474235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:18.6101369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:38:18.6124609Z 
2022-09-27T16:38:18.6124927Z Running tests...
2022-09-27T16:38:18.6125378Z ----------------------------------------------------------------------
2022-09-27T16:38:19.0160856Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:19.0161580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:19.0162370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:19.0163058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:19.0268722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:19.0571365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:19.0827526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:19.0828545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:19.0876454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:19.0885433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:19.0889090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:38:19.1173917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:19.1841728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:19.1844509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:38:19.1845331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:38:19.1904710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:38:19.7539439Z ok (12.849s)
2022-09-27T16:38:19.7539805Z 
2022-09-27T16:38:19.7540203Z ----------------------------------------------------------------------
2022-09-27T16:38:19.7540565Z Ran 1 test in 12.850s
2022-09-27T16:38:19.7540733Z 
2022-09-27T16:38:19.7540828Z OK
2022-09-27T16:38:19.7540973Z 
2022-09-27T16:38:19.7541113Z Generating XML reports...
2022-09-27T16:38:19.7600291Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163806.xml
2022-09-27T16:38:20.3181529Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:20.3381591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102397
2022-09-27T16:38:20.3386844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102398
2022-09-27T16:38:20.4132636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9swwk452
2022-09-27T16:38:20.4133662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9swwk452/_remote_module_non_scriptable.py
2022-09-27T16:38:20.4949548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby0idtj2
2022-09-27T16:38:20.4950667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby0idtj2/_remote_module_non_scriptable.py
2022-09-27T16:38:21.0060452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:21.0063689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:21.1252585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:21.1253540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:21.8224345Z ok (16.809s)
2022-09-27T16:38:21.8224596Z 
2022-09-27T16:38:21.8225005Z ----------------------------------------------------------------------
2022-09-27T16:38:21.8225355Z Ran 1 test in 16.809s
2022-09-27T16:38:21.8225507Z 
2022-09-27T16:38:21.8225619Z OK
2022-09-27T16:38:21.8225758Z 
2022-09-27T16:38:21.8225898Z Generating XML reports...
2022-09-27T16:38:21.8262213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163805.xml
2022-09-27T16:38:21.9683849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:21.9684346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:21.9686398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:21.9686893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:22.0223000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:22.0223503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:22.0224751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:22.0225236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:22.0566967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:22.0567436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:22.0569807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:22.0570290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:22.4935343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:22.5358056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:22.5358882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:22.5441308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:22.5448405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:22.5596541Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:22.5621485Z 
2022-09-27T16:38:22.5621937Z Running tests...
2022-09-27T16:38:22.5622462Z ----------------------------------------------------------------------
2022-09-27T16:38:22.6372387Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:23.9214735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoqd3eb_t
2022-09-27T16:38:23.9215362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoqd3eb_t/_remote_module_non_scriptable.py
2022-09-27T16:38:23.9352840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:23.9353321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:23.9356094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:23.9356594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:23.9947061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptl4xbi6a
2022-09-27T16:38:23.9948005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptl4xbi6a/_remote_module_non_scriptable.py
2022-09-27T16:38:24.2109343Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:24.2305524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102575
2022-09-27T16:38:24.2311998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102576
2022-09-27T16:38:24.4609084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:38:24.4632371Z 
2022-09-27T16:38:24.4632757Z Running tests...
2022-09-27T16:38:24.4633243Z ----------------------------------------------------------------------
2022-09-27T16:38:25.8538857Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:25.8540136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:25.8541642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:25.8542490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:25.8988343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:25.8989178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:25.8990825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:25.8991756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:26.1344671Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:26.1555284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102652
2022-09-27T16:38:26.1560160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102653
2022-09-27T16:38:26.4356383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:26.4357428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:26.4359135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:26.4360517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:26.4365460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:26.4366342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:26.6527854Z ok (8.040s)
2022-09-27T16:38:26.6528087Z 
2022-09-27T16:38:26.6528487Z ----------------------------------------------------------------------
2022-09-27T16:38:26.6528838Z Ran 1 test in 8.040s
2022-09-27T16:38:26.6528991Z 
2022-09-27T16:38:26.6529118Z OK
2022-09-27T16:38:26.6529257Z 
2022-09-27T16:38:26.6529394Z Generating XML reports...
2022-09-27T16:38:26.6568466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163818.xml
2022-09-27T16:38:27.8175239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:27.8175804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:27.8178087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:27.8178585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:27.8375879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:27.8376413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:27.8378642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:27.8379138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:27.9285336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5zb0_ew
2022-09-27T16:38:27.9286362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5zb0_ew/_remote_module_non_scriptable.py
2022-09-27T16:38:27.9550076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7b5owqsn
2022-09-27T16:38:27.9551248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7b5owqsn/_remote_module_non_scriptable.py
2022-09-27T16:38:27.9873087Z [1664296707.986692] [28fd6574525b:102575:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:38:27.9874397Z [1664296707.986692] [28fd6574525b:102575:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:38:27.9904858Z [1664296707.989987] [28fd6574525b:102576:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:38:27.9905854Z [1664296707.989987] [28fd6574525b:102576:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:38:28.3972516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:28.3973647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:28.3975071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:28.3976132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:28.3981376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:28.3982317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:28.8226455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:28.8227913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:28.8229125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:28.8230114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:28.8559087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:28.8560486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:28.8625738Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:38:28.8628917Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:38:29.3438171Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:38:29.3462005Z 
2022-09-27T16:38:29.3462341Z Running tests...
2022-09-27T16:38:29.3463196Z ----------------------------------------------------------------------
2022-09-27T16:38:29.4438542Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... ok (6.881s)
2022-09-27T16:38:29.4439140Z 
2022-09-27T16:38:29.4439810Z ----------------------------------------------------------------------
2022-09-27T16:38:29.4440667Z Ran 1 test in 6.882s
2022-09-27T16:38:29.4440982Z 
2022-09-27T16:38:29.4441139Z OK
2022-09-27T16:38:29.4441406Z 
2022-09-27T16:38:29.4441652Z Generating XML reports...
2022-09-27T16:38:29.4495686Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163822.xml
2022-09-27T16:38:29.8900996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzyz93z4
2022-09-27T16:38:29.8902523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzyz93z4/_remote_module_non_scriptable.py
2022-09-27T16:38:29.9043746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4j634xx6
2022-09-27T16:38:29.9045013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4j634xx6/_remote_module_non_scriptable.py
2022-09-27T16:38:30.4532099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:30.4533082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:30.5258623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:30.5259646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:31.1097189Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:31.1293414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102846
2022-09-27T16:38:31.1299065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102847
2022-09-27T16:38:31.2679160Z ok (6.804s)
2022-09-27T16:38:31.2679452Z 
2022-09-27T16:38:31.2679857Z ----------------------------------------------------------------------
2022-09-27T16:38:31.2680994Z Ran 1 test in 6.804s
2022-09-27T16:38:31.2681165Z 
2022-09-27T16:38:31.2681267Z OK
2022-09-27T16:38:31.2681403Z 
2022-09-27T16:38:31.2681521Z Generating XML reports...
2022-09-27T16:38:31.2716034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163824.xml
2022-09-27T16:38:31.7281419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:31.7281901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:31.7284893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:31.7285427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:32.2719860Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:32.2743108Z 
2022-09-27T16:38:32.2743364Z Running tests...
2022-09-27T16:38:32.2743827Z ----------------------------------------------------------------------
2022-09-27T16:38:32.2764622Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:38:32.2765861Z 
2022-09-27T16:38:32.2766286Z ----------------------------------------------------------------------
2022-09-27T16:38:32.2766646Z Ran 1 test in 0.002s
2022-09-27T16:38:32.2766812Z 
2022-09-27T16:38:32.2766926Z OK (skipped=1)
2022-09-27T16:38:32.2767087Z 
2022-09-27T16:38:32.2767196Z Generating XML reports...
2022-09-27T16:38:32.2801618Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163832.xml
2022-09-27T16:38:32.8182085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:32.8182614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:32.8183224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:32.8183695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:32.8189882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:32.8190361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:32.8193829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:32.8194325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:33.3306321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:33.3372699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:33.3374913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:33.3407752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:33.3415655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:33.3773853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:33.3774769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:33.3775967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:33.3776481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:33.4389890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:33.8835918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:38:33.8858186Z 
2022-09-27T16:38:33.8858490Z Running tests...
2022-09-27T16:38:33.8858970Z ----------------------------------------------------------------------
2022-09-27T16:38:34.2838224Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:34.2838941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:34.2839973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:34.2840700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:34.6483153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99zdecmo
2022-09-27T16:38:34.6485882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99zdecmo/_remote_module_non_scriptable.py
2022-09-27T16:38:34.8114947Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:34.8130507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jdtbiuv
2022-09-27T16:38:34.8133705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jdtbiuv/_remote_module_non_scriptable.py
2022-09-27T16:38:34.8138717Z 
2022-09-27T16:38:34.8138989Z Running tests...
2022-09-27T16:38:34.8139435Z ----------------------------------------------------------------------
2022-09-27T16:38:35.5178078Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:35.5380052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102993
2022-09-27T16:38:35.5384417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102994
2022-09-27T16:38:36.5123335Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:36.5333584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103063
2022-09-27T16:38:36.5338476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103064
2022-09-27T16:38:37.2442531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:37.2444166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:37.2447559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:37.2448666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:37.3185713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:37.3186888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:37.3191008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:37.3192647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:37.3428773Z ok (7.996s)
2022-09-27T16:38:37.3429119Z 
2022-09-27T16:38:37.3429708Z ----------------------------------------------------------------------
2022-09-27T16:38:37.3430363Z Ran 1 test in 7.997s
2022-09-27T16:38:37.3430691Z 
2022-09-27T16:38:37.3430872Z OK
2022-09-27T16:38:37.3431119Z 
2022-09-27T16:38:37.3431368Z Generating XML reports...
2022-09-27T16:38:37.3470736Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163829.xml
2022-09-27T16:38:37.8414497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:37.8415044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:37.8416121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:37.8416837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:37.8521867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:37.9428751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:38.2528789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:38.2529304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:38.2530335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:38.2530812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:38.2678948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:38.2679438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:38.2681868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:38.2682344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:38.8068088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:38.8739652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:38.8740511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:38.8777645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:38.8785417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:38.9753469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:39.2604230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49yels66
2022-09-27T16:38:39.2604856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49yels66/_remote_module_non_scriptable.py
2022-09-27T16:38:39.3147807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpjb0xf4
2022-09-27T16:38:39.3148665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpjb0xf4/_remote_module_non_scriptable.py
2022-09-27T16:38:39.4410513Z ok (4.627s)
2022-09-27T16:38:39.4410768Z 
2022-09-27T16:38:39.4411164Z ----------------------------------------------------------------------
2022-09-27T16:38:39.4411515Z Ran 1 test in 4.627s
2022-09-27T16:38:39.4411680Z 
2022-09-27T16:38:39.4411775Z OK
2022-09-27T16:38:39.4411911Z 
2022-09-27T16:38:39.4412045Z Generating XML reports...
2022-09-27T16:38:39.4446014Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163834.xml
2022-09-27T16:38:39.4706944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:39.4707437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:39.4708747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:39.4709236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:39.8734861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:39.8735408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:39.9485008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:39.9485841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:39.9813742Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:38:39.9836660Z 
2022-09-27T16:38:39.9837167Z Running tests...
2022-09-27T16:38:39.9837689Z ----------------------------------------------------------------------
2022-09-27T16:38:40.5499435Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... ok (6.664s)
2022-09-27T16:38:40.5499735Z 
2022-09-27T16:38:40.5500119Z ----------------------------------------------------------------------
2022-09-27T16:38:40.5500458Z Ran 1 test in 6.664s
2022-09-27T16:38:40.5500646Z 
2022-09-27T16:38:40.5500740Z OK
2022-09-27T16:38:40.5500877Z 
2022-09-27T16:38:40.5500988Z Generating XML reports...
2022-09-27T16:38:40.5536107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163833.xml
2022-09-27T16:38:41.5822819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:41.5823742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:41.5824787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:41.5825264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:41.6606592Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:41.6975435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103281
2022-09-27T16:38:41.7062650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103282
2022-09-27T16:38:42.0813404Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:42.0838592Z 
2022-09-27T16:38:42.0839020Z Running tests...
2022-09-27T16:38:42.0839528Z ----------------------------------------------------------------------
2022-09-27T16:38:42.0876385Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.004s)
2022-09-27T16:38:42.0876990Z 
2022-09-27T16:38:42.0877272Z ----------------------------------------------------------------------
2022-09-27T16:38:42.0877627Z Ran 1 test in 0.004s
2022-09-27T16:38:42.0877799Z 
2022-09-27T16:38:42.0877909Z OK (skipped=1)
2022-09-27T16:38:42.0878071Z 
2022-09-27T16:38:42.0878201Z Generating XML reports...
2022-09-27T16:38:42.0913329Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163842.xml
2022-09-27T16:38:42.6827268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:42.6828297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:42.6829485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:42.6830448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:43.2358008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:38:43.2379947Z 
2022-09-27T16:38:43.2380408Z Running tests...
2022-09-27T16:38:43.2380904Z ----------------------------------------------------------------------
2022-09-27T16:38:43.3650260Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:43.3650995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:43.3652406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:43.3653210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:43.3791417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:43.3791903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:43.3795499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:43.3795996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:43.8841677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:43.9007808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:43.9008643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:43.9044766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:43.9051746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:44.0023464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:44.0792551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:44.0793213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:44.0794532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:44.0795036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:44.6119262Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:44.6143700Z 
2022-09-27T16:38:44.6143931Z Running tests...
2022-09-27T16:38:44.6144380Z ----------------------------------------------------------------------
2022-09-27T16:38:44.6169420Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:38:44.6169960Z 
2022-09-27T16:38:44.6170241Z ----------------------------------------------------------------------
2022-09-27T16:38:44.6170571Z Ran 1 test in 0.003s
2022-09-27T16:38:44.6170735Z 
2022-09-27T16:38:44.6170846Z OK (skipped=1)
2022-09-27T16:38:44.6171006Z 
2022-09-27T16:38:44.6171132Z Generating XML reports...
2022-09-27T16:38:44.6206986Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163844.xml
2022-09-27T16:38:44.8575873Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:44.8780136Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103391
2022-09-27T16:38:44.8785402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103392
2022-09-27T16:38:45.2202929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp480s6reg
2022-09-27T16:38:45.2204262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp480s6reg/_remote_module_non_scriptable.py
2022-09-27T16:38:45.3174291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd86vdb3t
2022-09-27T16:38:45.3175377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd86vdb3t/_remote_module_non_scriptable.py
2022-09-27T16:38:46.5573651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:46.5574374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:46.5575321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:46.5576088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:46.6138611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:46.6139360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:46.6139992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:46.6140728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:46.6772557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:46.6773132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:46.6776029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:46.6776595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:47.1678863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:47.1679452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:47.1680796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:47.1681818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:47.1688095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:47.1688837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:47.1839091Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:47.1862698Z 
2022-09-27T16:38:47.1862866Z Running tests...
2022-09-27T16:38:47.1863590Z ----------------------------------------------------------------------
2022-09-27T16:38:47.1883432Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:38:47.1883989Z 
2022-09-27T16:38:47.1884486Z ----------------------------------------------------------------------
2022-09-27T16:38:47.1884851Z Ran 1 test in 0.002s
2022-09-27T16:38:47.1885151Z 
2022-09-27T16:38:47.1885352Z OK (skipped=1)
2022-09-27T16:38:47.1885555Z 
2022-09-27T16:38:47.1885665Z Generating XML reports...
2022-09-27T16:38:47.1921326Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163847.xml
2022-09-27T16:38:47.6199150Z ok (7.636s)
2022-09-27T16:38:47.6199563Z 
2022-09-27T16:38:47.6200932Z ----------------------------------------------------------------------
2022-09-27T16:38:47.6201625Z Ran 1 test in 7.636s
2022-09-27T16:38:47.6201940Z 
2022-09-27T16:38:47.6202112Z OK
2022-09-27T16:38:47.6202358Z 
2022-09-27T16:38:47.6202620Z Generating XML reports...
2022-09-27T16:38:47.6236923Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163839.xml
2022-09-27T16:38:48.7378100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf93397u0
2022-09-27T16:38:48.7378920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf93397u0/_remote_module_non_scriptable.py
2022-09-27T16:38:48.7453694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wu9eppp
2022-09-27T16:38:48.7456574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wu9eppp/_remote_module_non_scriptable.py
2022-09-27T16:38:49.2585137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:49.2585659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:49.2586966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:49.2587746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:49.3071367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:49.3071912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:49.3825282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:49.3825819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:49.7741209Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:49.7763485Z 
2022-09-27T16:38:49.7763768Z Running tests...
2022-09-27T16:38:49.7764235Z ----------------------------------------------------------------------
2022-09-27T16:38:49.7782166Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:38:49.7782505Z 
2022-09-27T16:38:49.7782769Z ----------------------------------------------------------------------
2022-09-27T16:38:49.7783113Z Ran 1 test in 0.002s
2022-09-27T16:38:49.7783284Z 
2022-09-27T16:38:49.7783397Z OK (skipped=1)
2022-09-27T16:38:49.7783556Z 
2022-09-27T16:38:49.7783688Z Generating XML reports...
2022-09-27T16:38:49.7818016Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163849.xml
2022-09-27T16:38:49.8887782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:49.8888303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:49.8889641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:49.8890118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:49.9898610Z ok (6.752s)
2022-09-27T16:38:49.9898817Z 
2022-09-27T16:38:49.9899182Z ----------------------------------------------------------------------
2022-09-27T16:38:49.9899512Z Ran 1 test in 6.752s
2022-09-27T16:38:49.9899683Z 
2022-09-27T16:38:49.9899786Z OK
2022-09-27T16:38:49.9899927Z 
2022-09-27T16:38:49.9900063Z Generating XML reports...
2022-09-27T16:38:49.9937086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163843.xml
2022-09-27T16:38:50.4082439Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:38:50.4106345Z 
2022-09-27T16:38:50.4106645Z Running tests...
2022-09-27T16:38:50.4107076Z ----------------------------------------------------------------------
2022-09-27T16:38:51.7218722Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:51.7219400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:51.7221488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:51.7221989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:52.0999475Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:52.1198646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103675
2022-09-27T16:38:52.1205711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103676
2022-09-27T16:38:52.2209088Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:52.2232030Z 
2022-09-27T16:38:52.2232526Z Running tests...
2022-09-27T16:38:52.2233364Z ----------------------------------------------------------------------
2022-09-27T16:38:52.2252142Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:38:52.2253165Z 
2022-09-27T16:38:52.2253734Z ----------------------------------------------------------------------
2022-09-27T16:38:52.2254337Z Ran 1 test in 0.002s
2022-09-27T16:38:52.2254656Z 
2022-09-27T16:38:52.2254851Z OK (skipped=1)
2022-09-27T16:38:52.2255144Z 
2022-09-27T16:38:52.2255376Z Generating XML reports...
2022-09-27T16:38:52.2273966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:52.2274962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:52.2279878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:52.2281389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:52.2289708Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163852.xml
2022-09-27T16:38:52.7243336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:38:52.7265280Z 
2022-09-27T16:38:52.7265561Z Running tests...
2022-09-27T16:38:52.7266010Z ----------------------------------------------------------------------
2022-09-27T16:38:53.7824549Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:53.7825422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:53.7826073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:53.7826558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:53.8218255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:53.8218762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:53.8221535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:53.8222032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:54.2416307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:54.2416806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:54.2419876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:54.2420583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:54.2908515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:54.3303350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:54.3304158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:54.3314411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:54.3321489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:54.3660897Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:38:54.4126138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103783
2022-09-27T16:38:54.4130986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103784
2022-09-27T16:38:54.4317654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:54.7646299Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:54.7670214Z 
2022-09-27T16:38:54.7670603Z Running tests...
2022-09-27T16:38:54.7671447Z ----------------------------------------------------------------------
2022-09-27T16:38:54.7690850Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:38:54.7691469Z 
2022-09-27T16:38:54.7691988Z ----------------------------------------------------------------------
2022-09-27T16:38:54.7692593Z Ran 1 test in 0.002s
2022-09-27T16:38:54.7692911Z 
2022-09-27T16:38:54.7693113Z OK (skipped=1)
2022-09-27T16:38:54.7693406Z 
2022-09-27T16:38:54.7693646Z Generating XML reports...
2022-09-27T16:38:54.7728314Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163854.xml
2022-09-27T16:38:55.7052891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7628hob
2022-09-27T16:38:55.7054098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7628hob/_remote_module_non_scriptable.py
2022-09-27T16:38:55.7929192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnkvnmsc
2022-09-27T16:38:55.7930378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnkvnmsc/_remote_module_non_scriptable.py
2022-09-27T16:38:56.0883656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:56.0884588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:56.0885838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:56.0886401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:56.1346958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:56.1347481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:56.1350201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:56.1350703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:56.6773660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:38:56.6774645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:38:56.6776090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:56.6777718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:38:56.6783201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:38:56.6783715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:38:56.8283997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:56.8284937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:56.8285994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:56.8286951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:57.3714813Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:38:57.3739660Z 
2022-09-27T16:38:57.3740107Z Running tests...
2022-09-27T16:38:57.3740624Z ----------------------------------------------------------------------
2022-09-27T16:38:57.3759419Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:38:57.3760119Z 
2022-09-27T16:38:57.3760926Z ----------------------------------------------------------------------
2022-09-27T16:38:57.3761277Z Ran 1 test in 0.002s
2022-09-27T16:38:57.3761445Z 
2022-09-27T16:38:57.3761538Z OK (skipped=1)
2022-09-27T16:38:57.3761762Z 
2022-09-27T16:38:57.3761998Z Generating XML reports...
2022-09-27T16:38:57.3791093Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:38:57.3797163Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163857.xml
2022-09-27T16:38:57.3853525Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:38:57.3934849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:57.3935399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:58.0341428Z ok (7.623s)
2022-09-27T16:38:58.0341628Z 
2022-09-27T16:38:58.0342033Z ----------------------------------------------------------------------
2022-09-27T16:38:58.0342568Z Ran 1 test in 7.623s
2022-09-27T16:38:58.0342810Z 
2022-09-27T16:38:58.0342906Z OK
2022-09-27T16:38:58.0343076Z 
2022-09-27T16:38:58.0343196Z Generating XML reports...
2022-09-27T16:38:58.0377800Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163850.xml
2022-09-27T16:38:58.1496406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8c4krvl
2022-09-27T16:38:58.1498187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8c4krvl/_remote_module_non_scriptable.py
2022-09-27T16:38:58.2306066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0qzntfe
2022-09-27T16:38:58.2307462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0qzntfe/_remote_module_non_scriptable.py
2022-09-27T16:38:58.8041984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:58.8042576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:58.8778295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:58.8778838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:38:59.5186879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:38:59.5187392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:38:59.5189472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:38:59.5189976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:38:59.5250407Z ok (6.798s)
2022-09-27T16:38:59.5250714Z 
2022-09-27T16:38:59.5251334Z ----------------------------------------------------------------------
2022-09-27T16:38:59.5251738Z Ran 1 test in 6.799s
2022-09-27T16:38:59.5252186Z 
2022-09-27T16:38:59.5252291Z OK
2022-09-27T16:38:59.5252428Z 
2022-09-27T16:38:59.5252564Z Generating XML reports...
2022-09-27T16:38:59.5289398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163852.xml
2022-09-27T16:39:00.0255535Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:00.0278520Z 
2022-09-27T16:39:00.0278916Z Running tests...
2022-09-27T16:39:00.0279504Z ----------------------------------------------------------------------
2022-09-27T16:39:00.1994100Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:00.1994967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:00.1997532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:00.1998038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:00.6972330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:39:00.6995195Z 
2022-09-27T16:39:00.6995636Z Running tests...
2022-09-27T16:39:00.6996227Z ----------------------------------------------------------------------
2022-09-27T16:39:01.5301138Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:01.5301813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:01.5302651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:01.5303155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:01.6423514Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:01.6630342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104042
2022-09-27T16:39:01.6635301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104043
2022-09-27T16:39:02.0316390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:39:02.0338051Z 
2022-09-27T16:39:02.0338467Z Running tests...
2022-09-27T16:39:02.0339283Z ----------------------------------------------------------------------
2022-09-27T16:39:02.2412981Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:02.2622872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104046
2022-09-27T16:39:02.2627784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104047
2022-09-27T16:39:03.3972866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:03.3973848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:03.3975051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:03.3975955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:03.4193849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:03.4194817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:03.4196562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:03.4197516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:03.6778232Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:03.6973241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104175
2022-09-27T16:39:03.6978988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104176
2022-09-27T16:39:03.9066751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:03.9398060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:03.9398895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:03.9472949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:03.9481517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:03.9484128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:39:03.9919605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:03.9920106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:03.9922818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:03.9923298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:04.0182867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:04.0183352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:04.0186353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:04.0186856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:04.0410517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:04.0414648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:39:04.0415375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:39:04.0499514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:39:04.5706966Z ok (4.543s)
2022-09-27T16:39:04.5707189Z 
2022-09-27T16:39:04.5707594Z ----------------------------------------------------------------------
2022-09-27T16:39:04.5707920Z Ran 1 test in 4.543s
2022-09-27T16:39:04.5708088Z 
2022-09-27T16:39:04.5708183Z OK
2022-09-27T16:39:04.5708320Z 
2022-09-27T16:39:04.5708732Z Generating XML reports...
2022-09-27T16:39:04.5743745Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163900.xml
2022-09-27T16:39:04.5758207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:04.5926146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:04.5926920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:04.5963945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:04.5971447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:04.6939351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:05.4156238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:05.4157110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:05.4158470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:05.4158963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:05.4276532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:05.4277016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:05.4280497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:05.4281221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:05.9775288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphqmv8taq
2022-09-27T16:39:05.9775912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphqmv8taq/_remote_module_non_scriptable.py
2022-09-27T16:39:06.0401914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:06.0402464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:06.0403218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:06.0403928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:06.0411493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:06.0411997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:06.0465318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hn77gtw
2022-09-27T16:39:06.0467843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hn77gtw/_remote_module_non_scriptable.py
2022-09-27T16:39:06.7687560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:06.7688074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:06.7690123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:06.7690619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:07.2789414Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:39:07.2790259Z   warnings.warn(
2022-09-27T16:39:07.2791328Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:39:07.2792080Z   warnings.warn(
2022-09-27T16:39:07.2865204Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:07.2889129Z 
2022-09-27T16:39:07.2889928Z Running tests...
2022-09-27T16:39:07.2890908Z ----------------------------------------------------------------------
2022-09-27T16:39:07.2915334Z   test_get_future (__main__.TestDistBackendWithSpawn) ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-09-27T16:39:07.2915717Z 
2022-09-27T16:39:07.2916178Z ----------------------------------------------------------------------
2022-09-27T16:39:07.2916665Z Ran 1 test in 0.003s
2022-09-27T16:39:07.2917169Z 
2022-09-27T16:39:07.2917281Z OK (skipped=1)
2022-09-27T16:39:07.2917421Z 
2022-09-27T16:39:07.2917550Z Generating XML reports...
2022-09-27T16:39:07.2952142Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163907.xml
2022-09-27T16:39:07.4637736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0s1v9gq
2022-09-27T16:39:07.4638630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0s1v9gq/_remote_module_non_scriptable.py
2022-09-27T16:39:07.4697772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1s0l77f9
2022-09-27T16:39:07.4699534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1s0l77f9/_remote_module_non_scriptable.py
2022-09-27T16:39:08.0193546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:08.0194421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:08.0903087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:08.0904030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:08.3760979Z ok (7.676s)
2022-09-27T16:39:08.3762052Z 
2022-09-27T16:39:08.3763020Z ----------------------------------------------------------------------
2022-09-27T16:39:08.3763699Z Ran 1 test in 7.676s
2022-09-27T16:39:08.3764045Z 
2022-09-27T16:39:08.3764202Z OK
2022-09-27T16:39:08.3764467Z 
2022-09-27T16:39:08.3764716Z Generating XML reports...
2022-09-27T16:39:08.3808267Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163900.xml
2022-09-27T16:39:08.7094113Z ok (6.675s)
2022-09-27T16:39:08.7094369Z 
2022-09-27T16:39:08.7094742Z ----------------------------------------------------------------------
2022-09-27T16:39:08.7095085Z Ran 1 test in 6.676s
2022-09-27T16:39:08.7095263Z 
2022-09-27T16:39:08.7095356Z OK
2022-09-27T16:39:08.7095490Z 
2022-09-27T16:39:08.7095610Z Generating XML reports...
2022-09-27T16:39:08.7152904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163902.xml
2022-09-27T16:39:09.3819276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:09.3819796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:09.3822617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:09.3823104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:09.9967449Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:09.9990058Z 
2022-09-27T16:39:09.9990500Z Running tests...
2022-09-27T16:39:09.9991006Z ----------------------------------------------------------------------
2022-09-27T16:39:10.5424678Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:10.5425338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:10.5427356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:10.5427835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:10.9020631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:10.9021301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:10.9023807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:10.9024309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:11.0845443Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:39:11.0878700Z 
2022-09-27T16:39:11.0878868Z Running tests...
2022-09-27T16:39:11.0879599Z ----------------------------------------------------------------------
2022-09-27T16:39:11.4163786Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:39:11.4186667Z 
2022-09-27T16:39:11.4187112Z Running tests...
2022-09-27T16:39:11.4187569Z ----------------------------------------------------------------------
2022-09-27T16:39:11.6105190Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:11.6312229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104445
2022-09-27T16:39:11.6316995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104446
2022-09-27T16:39:12.7627141Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:12.7830426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104510
2022-09-27T16:39:12.7835777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104511
2022-09-27T16:39:13.0541743Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:13.0738603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104513
2022-09-27T16:39:13.0744838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104514
2022-09-27T16:39:13.2630283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:13.2630790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:13.2632706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:13.2656219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:13.2657136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:13.2657604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:13.2660434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:13.2661168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:13.8900941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:13.9078782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:13.9079677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:13.9103494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:13.9110913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:14.0092119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:14.4949028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:14.4949723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:14.4950389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:14.4951082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:14.5098642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:14.5099487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:14.5102119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:14.5102788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:14.5386478Z ok (4.539s)
2022-09-27T16:39:14.5386693Z 
2022-09-27T16:39:14.5387036Z ----------------------------------------------------------------------
2022-09-27T16:39:14.5387388Z Ran 1 test in 4.540s
2022-09-27T16:39:14.5387554Z 
2022-09-27T16:39:14.5387647Z OK
2022-09-27T16:39:14.5387765Z 
2022-09-27T16:39:14.5387902Z Generating XML reports...
2022-09-27T16:39:14.5423934Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163909.xml
2022-09-27T16:39:14.7792396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:14.7792931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:14.7794520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:14.7794991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:14.8190706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:14.8191205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:14.8193331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:14.8193843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:15.0277116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:15.0383003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:15.0383818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:15.0479313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:15.0486917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:15.1398405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:15.3712467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:15.3713261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:15.3714058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:15.3714780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:15.3821589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:15.4728893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:16.5134474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87fquv19
2022-09-27T16:39:16.5135536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87fquv19/_remote_module_non_scriptable.py
2022-09-27T16:39:16.5593057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5qcu75l
2022-09-27T16:39:16.5594224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5qcu75l/_remote_module_non_scriptable.py
2022-09-27T16:39:16.6295358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:16.6296855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:16.6298050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:16.6298977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:16.7705171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj10si1wm
2022-09-27T16:39:16.7706335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj10si1wm/_remote_module_non_scriptable.py
2022-09-27T16:39:16.8751215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprm9mw3yb
2022-09-27T16:39:16.8751831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprm9mw3yb/_remote_module_non_scriptable.py
2022-09-27T16:39:16.9229823Z STAGE:2022-09-27 16:39:16 104511:104511 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:16.9244021Z STAGE:2022-09-27 16:39:16 104510:104510 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:17.1813013Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:17.1835641Z 
2022-09-27T16:39:17.1835955Z Running tests...
2022-09-27T16:39:17.1836404Z ----------------------------------------------------------------------
2022-09-27T16:39:17.4102640Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:17.4103290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:17.4964847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:17.4965448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:17.5183704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:17.5206643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:17.5281350Z STAGE:2022-09-27 16:39:17 104510:104510 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:17.5282056Z STAGE:2022-09-27 16:39:17 104511:104511 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:17.5588444Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:39:17.5590110Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:39:17.5637003Z STAGE:2022-09-27 16:39:17 104511:104511 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:17.5637612Z STAGE:2022-09-27 16:39:17 104510:104510 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:17.5679084Z STAGE:2022-09-27 16:39:17 104511:104511 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:17.5685158Z STAGE:2022-09-27 16:39:17 104510:104510 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:18.1858873Z ok (6.767s)
2022-09-27T16:39:18.1859098Z 
2022-09-27T16:39:18.1859499Z ----------------------------------------------------------------------
2022-09-27T16:39:18.1859844Z Ran 1 test in 6.767s
2022-09-27T16:39:18.1860012Z 
2022-09-27T16:39:18.1860098Z OK
2022-09-27T16:39:18.1860235Z 
2022-09-27T16:39:18.1860372Z Generating XML reports...
2022-09-27T16:39:18.1895856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163911.xml
2022-09-27T16:39:18.2961879Z ok (7.208s)
2022-09-27T16:39:18.2962101Z 
2022-09-27T16:39:18.2962619Z ----------------------------------------------------------------------
2022-09-27T16:39:18.2963075Z Ran 1 test in 7.208s
2022-09-27T16:39:18.2963254Z 
2022-09-27T16:39:18.2963367Z OK
2022-09-27T16:39:18.2963505Z 
2022-09-27T16:39:18.2963644Z Generating XML reports...
2022-09-27T16:39:18.2999119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163911.xml
2022-09-27T16:39:18.9384313Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:18.9579627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104751
2022-09-27T16:39:18.9585875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104752
2022-09-27T16:39:20.4268180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:20.4269152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:20.4270349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:20.4271352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:20.6199338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:20.6200721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:20.6201943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:20.6202896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:20.6967939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:20.6968825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:20.6969873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:20.6970711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:20.7251258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:20.7252271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:20.7254158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:20.7255094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:20.9119252Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:39:20.9142166Z 
2022-09-27T16:39:20.9142521Z Running tests...
2022-09-27T16:39:20.9142967Z ----------------------------------------------------------------------
2022-09-27T16:39:21.1590583Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:39:21.1612749Z 
2022-09-27T16:39:21.1613000Z Running tests...
2022-09-27T16:39:21.1613453Z ----------------------------------------------------------------------
2022-09-27T16:39:21.2110696Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:21.2555297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:21.2556129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:21.2617600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:21.2625547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:21.2628699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:39:21.3570451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:21.3573433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:39:21.3574229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:39:21.3644798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:39:21.8658299Z ok (4.682s)
2022-09-27T16:39:21.8658513Z 
2022-09-27T16:39:21.8658885Z ----------------------------------------------------------------------
2022-09-27T16:39:21.8659237Z Ran 1 test in 4.682s
2022-09-27T16:39:21.8659405Z 
2022-09-27T16:39:21.8659504Z OK
2022-09-27T16:39:21.8659642Z 
2022-09-27T16:39:21.8659772Z Generating XML reports...
2022-09-27T16:39:21.8695367Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163917.xml
2022-09-27T16:39:22.5401995Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:22.5596846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104887
2022-09-27T16:39:22.5603004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104888
2022-09-27T16:39:22.7524915Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:22.7718832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104890
2022-09-27T16:39:22.7725107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104891
2022-09-27T16:39:24.2764979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:24.2765488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:24.2768788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:24.2769315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:24.3150963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:24.3151456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:24.3152867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:24.3153351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:24.4543857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:24.4544361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:24.4546530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:24.4547046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:24.6305654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:24.6306496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:24.6307680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:24.6308162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:24.6698421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:24.6698898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:24.6700989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:24.6701488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:24.7845623Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:24.7868600Z 
2022-09-27T16:39:24.7868916Z Running tests...
2022-09-27T16:39:24.7869622Z ----------------------------------------------------------------------
2022-09-27T16:39:24.9920943Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:24.9921583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:24.9922347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:24.9923045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:24.9929953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:24.9930675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:25.1505702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:25.1850839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:25.1851694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:25.1911554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:25.1918861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:25.2866104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:26.4445728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9dmicmd
2022-09-27T16:39:26.4447165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9dmicmd/_remote_module_non_scriptable.py
2022-09-27T16:39:26.4608944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sgoj6x0
2022-09-27T16:39:26.4610226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sgoj6x0/_remote_module_non_scriptable.py
2022-09-27T16:39:26.5181742Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:26.5393298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105102
2022-09-27T16:39:26.5398328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105103
2022-09-27T16:39:26.5821132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s848pmu
2022-09-27T16:39:26.5822062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s848pmu/_remote_module_non_scriptable.py
2022-09-27T16:39:26.6289426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxogzsy9x
2022-09-27T16:39:26.6290332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxogzsy9x/_remote_module_non_scriptable.py
2022-09-27T16:39:26.9781903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:26.9782443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:26.9922195Z STAGE:2022-09-27 16:39:26 104891:104891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:26.9944585Z STAGE:2022-09-27 16:39:26 104890:104890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:27.0576193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:27.0577163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:27.6467938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:27.6469862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:27.6542823Z STAGE:2022-09-27 16:39:27 104890:104890 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:27.6545042Z STAGE:2022-09-27 16:39:27 104891:104891 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:27.7033678Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:39:27.7035421Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:39:27.7072278Z STAGE:2022-09-27 16:39:27 104890:104890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:27.7072862Z STAGE:2022-09-27 16:39:27 104891:104891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:27.7114830Z STAGE:2022-09-27 16:39:27 104890:104890 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:27.7115721Z STAGE:2022-09-27 16:39:27 104891:104891 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:27.7722293Z ok (6.858s)
2022-09-27T16:39:27.7722489Z 
2022-09-27T16:39:27.7722894Z ----------------------------------------------------------------------
2022-09-27T16:39:27.7723262Z Ran 1 test in 6.858s
2022-09-27T16:39:27.7723435Z 
2022-09-27T16:39:27.7723528Z OK
2022-09-27T16:39:27.7723674Z 
2022-09-27T16:39:27.7723808Z Generating XML reports...
2022-09-27T16:39:27.7758673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163920.xml
2022-09-27T16:39:28.2845732Z ok (7.123s)
2022-09-27T16:39:28.2845936Z 
2022-09-27T16:39:28.2846289Z ----------------------------------------------------------------------
2022-09-27T16:39:28.2846672Z Ran 1 test in 7.123s
2022-09-27T16:39:28.2846841Z 
2022-09-27T16:39:28.2846947Z OK
2022-09-27T16:39:28.2847078Z 
2022-09-27T16:39:28.2847211Z Generating XML reports...
2022-09-27T16:39:28.2881491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163921.xml
2022-09-27T16:39:28.3193782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:28.3194591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:28.3195784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:28.3196265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:28.3626486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:28.3626996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:28.3629160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:28.3629659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:28.8657198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:28.8813380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:28.8814205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:28.8860335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:28.8867003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:28.8869803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:39:28.9882910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:28.9884635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:39:28.9886426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:39:28.9888246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:39:29.4470512Z ok (4.660s)
2022-09-27T16:39:29.4470749Z 
2022-09-27T16:39:29.4471142Z ----------------------------------------------------------------------
2022-09-27T16:39:29.4471724Z Ran 1 test in 4.660s
2022-09-27T16:39:29.4471920Z 
2022-09-27T16:39:29.4472021Z OK
2022-09-27T16:39:29.4472159Z 
2022-09-27T16:39:29.4472302Z Generating XML reports...
2022-09-27T16:39:29.4508291Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163924.xml
2022-09-27T16:39:29.9841505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:29.9842308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:29.9844412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:29.9844920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:30.4702125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:39:30.4724422Z 
2022-09-27T16:39:30.4724683Z Running tests...
2022-09-27T16:39:30.4725127Z ----------------------------------------------------------------------
2022-09-27T16:39:30.5015994Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:30.5016601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:30.5021057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:30.5021551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:31.0062518Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:39:31.0087190Z 
2022-09-27T16:39:31.0087566Z Running tests...
2022-09-27T16:39:31.0088446Z ----------------------------------------------------------------------
2022-09-27T16:39:31.5296068Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:31.5297256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:31.5298455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:31.5299413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:32.0324680Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:32.0346334Z 
2022-09-27T16:39:32.0346969Z Running tests...
2022-09-27T16:39:32.0347479Z ----------------------------------------------------------------------
2022-09-27T16:39:32.0374297Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-09-27T16:39:32.0374633Z 
2022-09-27T16:39:32.0375168Z ----------------------------------------------------------------------
2022-09-27T16:39:32.0375642Z Ran 1 test in 0.003s
2022-09-27T16:39:32.0375813Z 
2022-09-27T16:39:32.0375935Z OK (skipped=1)
2022-09-27T16:39:32.0376097Z 
2022-09-27T16:39:32.0376227Z Generating XML reports...
2022-09-27T16:39:32.0410611Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163932.xml
2022-09-27T16:39:32.0566561Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:32.0770658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105291
2022-09-27T16:39:32.0776041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105292
2022-09-27T16:39:32.6025633Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:32.6226825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105295
2022-09-27T16:39:32.6232547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105296
2022-09-27T16:39:33.7976816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:33.7977578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:33.7978674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:33.7979180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:33.8012838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:33.8013337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:33.8016570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:33.8017040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:34.1126219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:34.1126706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:34.1128783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:34.1129281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:34.3222772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:34.3223329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:34.3224298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:34.3224980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:34.3330873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:34.3546288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:34.3546777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:34.3549279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:34.3549772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:34.3998999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:34.3999873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:34.4001159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:34.4001945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:34.4236999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:34.6259625Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:34.6283729Z 
2022-09-27T16:39:34.6284154Z Running tests...
2022-09-27T16:39:34.6284637Z ----------------------------------------------------------------------
2022-09-27T16:39:34.9073866Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:34.9337647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:34.9338507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:34.9378747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:34.9385994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:35.0352418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:35.7675394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpda82ffom
2022-09-27T16:39:35.7676741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpda82ffom/_remote_module_non_scriptable.py
2022-09-27T16:39:35.8139503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3t_e891
2022-09-27T16:39:35.8140150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3t_e891/_remote_module_non_scriptable.py
2022-09-27T16:39:36.2982731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeogbq98e
2022-09-27T16:39:36.2984356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeogbq98e/_remote_module_non_scriptable.py
2022-09-27T16:39:36.3154996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:36.3155584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:36.3516095Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:36.3733565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105513
2022-09-27T16:39:36.3737748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105514
2022-09-27T16:39:36.3828820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:36.3829726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:36.4012445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_r3jh5dy
2022-09-27T16:39:36.4013817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_r3jh5dy/_remote_module_non_scriptable.py
2022-09-27T16:39:36.9895991Z ok (6.517s)
2022-09-27T16:39:36.9896224Z 
2022-09-27T16:39:36.9896590Z ----------------------------------------------------------------------
2022-09-27T16:39:36.9896941Z Ran 1 test in 6.517s
2022-09-27T16:39:36.9897110Z 
2022-09-27T16:39:36.9897217Z OK
2022-09-27T16:39:36.9897357Z 
2022-09-27T16:39:36.9897959Z Generating XML reports...
2022-09-27T16:39:36.9933191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163930.xml
2022-09-27T16:39:38.0818450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:38.0819433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:38.0821010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:38.0822098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:38.0824450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:38.0825348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:38.0829537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:38.0830484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:38.2359549Z ok (7.227s)
2022-09-27T16:39:38.2359932Z 
2022-09-27T16:39:38.2361175Z ----------------------------------------------------------------------
2022-09-27T16:39:38.2361836Z Ran 1 test in 7.227s
2022-09-27T16:39:38.2362179Z 
2022-09-27T16:39:38.2362357Z OK
2022-09-27T16:39:38.2362617Z 
2022-09-27T16:39:38.2362870Z Generating XML reports...
2022-09-27T16:39:38.2399785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163931.xml
2022-09-27T16:39:38.6103664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:38.6128436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:38.6129499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:38.6205513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:38.6211961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:38.7142078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:39.1826533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:39.1827060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:39.1830249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:39.1830753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:39.7660045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:39:39.7684522Z 
2022-09-27T16:39:39.7684945Z Running tests...
2022-09-27T16:39:39.7685483Z ----------------------------------------------------------------------
2022-09-27T16:39:40.1313062Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... [1664296780.130743] [28fd6574525b:105514:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:39:40.1314089Z [1664296780.130743] [28fd6574525b:105514:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:39:40.1321259Z [1664296780.131799] [28fd6574525b:105513:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:39:40.1321730Z [1664296780.131799] [28fd6574525b:105513:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:39:40.3789179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:40.3789697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:40.3794816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:40.3795313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:40.9613426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:39:40.9637282Z 
2022-09-27T16:39:40.9637742Z Running tests...
2022-09-27T16:39:40.9638255Z ----------------------------------------------------------------------
2022-09-27T16:39:40.9840685Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... ok (6.355s)
2022-09-27T16:39:40.9841020Z 
2022-09-27T16:39:40.9841350Z ----------------------------------------------------------------------
2022-09-27T16:39:40.9841694Z Ran 1 test in 6.356s
2022-09-27T16:39:40.9841861Z 
2022-09-27T16:39:40.9841957Z OK
2022-09-27T16:39:40.9842075Z 
2022-09-27T16:39:40.9842202Z Generating XML reports...
2022-09-27T16:39:40.9879304Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163934.xml
2022-09-27T16:39:41.4543877Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:41.4749320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105664
2022-09-27T16:39:41.4755856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105665
2022-09-27T16:39:42.6247219Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:42.6444278Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105760
2022-09-27T16:39:42.6451459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105761
2022-09-27T16:39:43.1244360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:43.1244886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:43.1246195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:43.1246690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:43.1549612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:43.1550101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:43.1553147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:43.1553615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:43.1685208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:43.1685977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:43.1689596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:43.1690071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:43.7297943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:43.7298754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:43.7299559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:43.7300268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:43.7305833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:43.7306780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:43.7358746Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:43.7383346Z 
2022-09-27T16:39:43.7383507Z Running tests...
2022-09-27T16:39:43.7384195Z ----------------------------------------------------------------------
2022-09-27T16:39:44.2831548Z   test_isend (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:44.2832341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:44.2833374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:44.2833883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:44.2895928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:44.2896413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:44.2899501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:44.2900014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:44.8278674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:44.8279240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:44.8280023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:44.8281046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:44.8287284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:44.8288250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:45.1982203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoenmga23
2022-09-27T16:39:45.1983079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoenmga23/_remote_module_non_scriptable.py
2022-09-27T16:39:45.2070232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesyy6vbi
2022-09-27T16:39:45.2071238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesyy6vbi/_remote_module_non_scriptable.py
2022-09-27T16:39:45.4668259Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:45.4877818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105879
2022-09-27T16:39:45.4883380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105880
2022-09-27T16:39:45.7537148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:45.7538249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:45.8222218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:45.8223264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:46.2922156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpky3omhal
2022-09-27T16:39:46.2923115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpky3omhal/_remote_module_non_scriptable.py
2022-09-27T16:39:46.2998468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp444dmzl8
2022-09-27T16:39:46.3000615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp444dmzl8/_remote_module_non_scriptable.py
2022-09-27T16:39:46.3980881Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:39:46.3981745Z   warnings.warn(
2022-09-27T16:39:46.3982830Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:39:46.3983570Z   warnings.warn(
2022-09-27T16:39:46.5882970Z ok (6.820s)
2022-09-27T16:39:46.5883181Z 
2022-09-27T16:39:46.5883740Z ----------------------------------------------------------------------
2022-09-27T16:39:46.5884155Z Ran 1 test in 6.820s
2022-09-27T16:39:46.5884322Z 
2022-09-27T16:39:46.5884419Z OK
2022-09-27T16:39:46.5884553Z 
2022-09-27T16:39:46.5884689Z Generating XML reports...
2022-09-27T16:39:46.5919342Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163939.xml
2022-09-27T16:39:46.9275447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:46.9275993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:46.9331821Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:39:46.9333688Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:39:46.9439119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:46.9439626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:46.9507484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:46.9507976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:47.1925249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:47.1925757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:47.1928323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:47.1929127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:47.2260279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:47.2261243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:47.2262330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:47.2262824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:47.5563811Z ok (6.592s)
2022-09-27T16:39:47.5564040Z 
2022-09-27T16:39:47.5564656Z ----------------------------------------------------------------------
2022-09-27T16:39:47.5564987Z Ran 1 test in 6.593s
2022-09-27T16:39:47.5565164Z 
2022-09-27T16:39:47.5565257Z OK
2022-09-27T16:39:47.5565395Z 
2022-09-27T16:39:47.5565537Z Generating XML reports...
2022-09-27T16:39:47.5600139Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163940.xml
2022-09-27T16:39:47.7428431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:47.7581133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:47.7584116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:47.7636327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:47.7729785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:47.8656808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:48.8623692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:48.8624249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:48.8626594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:48.8627105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:49.2764015Z [1664296789.275883] [28fd6574525b:105880:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:39:49.2764553Z [1664296789.275883] [28fd6574525b:105880:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:39:49.3453274Z [1664296789.344812] [28fd6574525b:105879:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:39:49.3453818Z [1664296789.344812] [28fd6574525b:105879:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:39:49.3675076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:39:49.3698226Z 
2022-09-27T16:39:49.3698459Z Running tests...
2022-09-27T16:39:49.3699137Z ----------------------------------------------------------------------
2022-09-27T16:39:49.7404510Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:49.7405530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:49.7406891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:49.7407705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:50.1993065Z ok (6.460s)
2022-09-27T16:39:50.1993285Z 
2022-09-27T16:39:50.1994037Z ----------------------------------------------------------------------
2022-09-27T16:39:50.1994384Z Ran 1 test in 6.461s
2022-09-27T16:39:50.1994540Z 
2022-09-27T16:39:50.1994634Z OK
2022-09-27T16:39:50.1994770Z 
2022-09-27T16:39:50.1994905Z Generating XML reports...
2022-09-27T16:39:50.2030133Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163943.xml
2022-09-27T16:39:50.2660933Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:39:50.2685845Z 
2022-09-27T16:39:50.2686201Z Running tests...
2022-09-27T16:39:50.2686648Z ----------------------------------------------------------------------
2022-09-27T16:39:51.0482549Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:51.0680992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106039
2022-09-27T16:39:51.0687450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106040
2022-09-27T16:39:52.0000865Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:52.0215157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106135
2022-09-27T16:39:52.0219976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106136
2022-09-27T16:39:52.3777575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:52.3778120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:52.3780414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:52.3780918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:52.8240955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:52.8241498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:52.8242103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:52.8242711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:52.8244198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:52.8244696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:52.8245293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:52.8245771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:52.9121246Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:39:52.9146045Z 
2022-09-27T16:39:52.9146380Z Running tests...
2022-09-27T16:39:52.9146829Z ----------------------------------------------------------------------
2022-09-27T16:39:53.3744305Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:53.3744952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:53.3745755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:53.3746461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:53.3753531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:53.3754031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:53.6993871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:53.6994765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:53.6996120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:53.6996649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:53.7080703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:53.7081193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:53.7084424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:53.7084938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:54.2408675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:54.2538579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:54.2539424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:54.2612482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:54.2619106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:54.3554316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:54.6029402Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:39:54.6240326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106224
2022-09-27T16:39:54.6245387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106225
2022-09-27T16:39:54.7676237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph63q4el2
2022-09-27T16:39:54.7676874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph63q4el2/_remote_module_non_scriptable.py
2022-09-27T16:39:54.8068353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpho3loonf
2022-09-27T16:39:54.8069387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpho3loonf/_remote_module_non_scriptable.py
2022-09-27T16:39:55.2615171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.2615760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.3251933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.3263253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.3726905Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:39:55.3751177Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:39:55.4452542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.4453112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.4655666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.4656250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.4800498Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:39:55.4801320Z   warnings.warn(
2022-09-27T16:39:55.4802399Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:39:55.4803134Z   warnings.warn(
2022-09-27T16:39:55.4885529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.4886075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.5076458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.5076997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:55.6746334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ixzzt0k
2022-09-27T16:39:55.6747000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ixzzt0k/_remote_module_non_scriptable.py
2022-09-27T16:39:55.7817806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp694tj8uk
2022-09-27T16:39:55.7818886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp694tj8uk/_remote_module_non_scriptable.py
2022-09-27T16:39:55.8690017Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:39:55.8690810Z   warnings.warn(
2022-09-27T16:39:55.8692189Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:39:55.8692963Z   warnings.warn(
2022-09-27T16:39:56.0813300Z ok (6.711s)
2022-09-27T16:39:56.0813513Z 
2022-09-27T16:39:56.0813902Z ----------------------------------------------------------------------
2022-09-27T16:39:56.0814257Z Ran 1 test in 6.711s
2022-09-27T16:39:56.0814425Z 
2022-09-27T16:39:56.0814523Z OK
2022-09-27T16:39:56.0814642Z 
2022-09-27T16:39:56.0814775Z Generating XML reports...
2022-09-27T16:39:56.0849474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163949.xml
2022-09-27T16:39:56.3385404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:56.3385984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:56.3387102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:56.3387887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:56.3537633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:56.3538163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:39:56.3664508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:56.3664983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:56.3668069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:56.3668571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:56.8981237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:39:56.8982109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:56.8982672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:39:56.8983330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:39:56.8988487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:39:56.8989607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:39:56.9083402Z STAGE:2022-09-27 16:39:56 106225:106225 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:56.9084044Z STAGE:2022-09-27 16:39:56 106224:106224 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:39:56.9326098Z ok (6.664s)
2022-09-27T16:39:56.9326472Z 
2022-09-27T16:39:56.9326888Z ----------------------------------------------------------------------
2022-09-27T16:39:56.9327226Z Ran 1 test in 6.664s
2022-09-27T16:39:56.9327393Z 
2022-09-27T16:39:56.9327485Z OK
2022-09-27T16:39:56.9327619Z 
2022-09-27T16:39:56.9327733Z Generating XML reports...
2022-09-27T16:39:56.9363783Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163950.xml
2022-09-27T16:39:58.3774664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:58.3775219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:58.3778205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:58.3778740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:58.6484837Z [1664296798.647994] [28fd6574525b:106225:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:39:58.6485378Z [1664296798.647994] [28fd6574525b:106225:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:39:58.6880894Z [1664296798.687603] [28fd6574525b:106224:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:39:58.6881432Z [1664296798.687603] [28fd6574525b:106224:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:39:58.8946858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:39:58.8971728Z 
2022-09-27T16:39:58.8971847Z Running tests...
2022-09-27T16:39:58.8972765Z ----------------------------------------------------------------------
2022-09-27T16:39:59.1212910Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:39:59.1213886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:39:59.1214848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:39:59.1215321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:39:59.1350870Z STAGE:2022-09-27 16:39:59 106224:106224 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:59.1352382Z STAGE:2022-09-27 16:39:59 106225:106225 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:39:59.6361190Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:39:59.6383634Z 
2022-09-27T16:39:59.6384004Z Running tests...
2022-09-27T16:39:59.6384433Z ----------------------------------------------------------------------
2022-09-27T16:40:00.4375327Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... ok (7.523s)
2022-09-27T16:40:00.4375661Z 
2022-09-27T16:40:00.4376067Z ----------------------------------------------------------------------
2022-09-27T16:40:00.4376409Z Ran 1 test in 7.523s
2022-09-27T16:40:00.4376577Z 
2022-09-27T16:40:00.4376676Z OK
2022-09-27T16:40:00.4376814Z 
2022-09-27T16:40:00.4376923Z Generating XML reports...
2022-09-27T16:40:00.4411766Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163952.xml
2022-09-27T16:40:00.6358653Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:00.6567635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106389
2022-09-27T16:40:00.6573242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106390
2022-09-27T16:40:01.3341330Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:01.4144258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106455
2022-09-27T16:40:01.4251316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106456
2022-09-27T16:40:02.4420953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:02.4421661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:02.4422933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:02.4423422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:02.4651087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:02.4651810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:02.4654066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:02.4654581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:02.6516147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:02.6516636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:02.6518636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:02.6519123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:02.9949813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:02.9950373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:02.9951218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:02.9951910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:02.9958071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:02.9958579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:03.1361759Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:03.1384152Z 
2022-09-27T16:40:03.1384607Z Running tests...
2022-09-27T16:40:03.1385097Z ----------------------------------------------------------------------
2022-09-27T16:40:03.1898341Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:03.1899230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:03.1900261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:03.1900747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:03.1924797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:03.1925277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:03.1929016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:03.1929491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:03.7359933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:03.7587449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:03.7588304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:03.7666111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:03.7672568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:03.8601547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:04.4264819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8atow57c
2022-09-27T16:40:04.4265422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8atow57c/_remote_module_non_scriptable.py
2022-09-27T16:40:04.4322611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0b89nez
2022-09-27T16:40:04.4325497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0b89nez/_remote_module_non_scriptable.py
2022-09-27T16:40:04.8047120Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:04.8251825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106575
2022-09-27T16:40:04.8256985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106576
2022-09-27T16:40:05.1085900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbsrimu2
2022-09-27T16:40:05.1086733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbsrimu2/_remote_module_non_scriptable.py
2022-09-27T16:40:05.2075799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiaaap5fs
2022-09-27T16:40:05.2076629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiaaap5fs/_remote_module_non_scriptable.py
2022-09-27T16:40:05.7936143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:05.7939243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:05.8041117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:05.8044323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:05.8693066Z ok (6.972s)
2022-09-27T16:40:05.8693282Z 
2022-09-27T16:40:05.8693818Z ----------------------------------------------------------------------
2022-09-27T16:40:05.8694449Z Ran 1 test in 6.972s
2022-09-27T16:40:05.8694633Z 
2022-09-27T16:40:05.8694728Z OK
2022-09-27T16:40:05.8694867Z 
2022-09-27T16:40:05.8694989Z Generating XML reports...
2022-09-27T16:40:05.8731153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163958.xml
2022-09-27T16:40:06.4373655Z ok (6.799s)
2022-09-27T16:40:06.4373864Z 
2022-09-27T16:40:06.4374364Z ----------------------------------------------------------------------
2022-09-27T16:40:06.4374834Z Ran 1 test in 6.799s
2022-09-27T16:40:06.4375015Z 
2022-09-27T16:40:06.4375124Z OK
2022-09-27T16:40:06.4375265Z 
2022-09-27T16:40:06.4375386Z Generating XML reports...
2022-09-27T16:40:06.4429122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163959.xml
2022-09-27T16:40:06.4959971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:06.4960997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:06.4962255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:06.4962728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:06.5202994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:06.5203511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:06.5209068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:06.5209550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:07.0390506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:07.0496794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:07.0498457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:07.0592439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:07.0599634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:07.0666022Z STAGE:2022-09-27 16:40:07 106576:106576 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:40:07.1598672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:07.1701129Z STAGE:2022-09-27 16:40:07 106575:106575 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:40:08.1756342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:08.1756842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:08.1759344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:08.1759839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:08.6595602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:08.6596134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:08.6597055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:08.6597835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:08.6997424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:40:08.7021825Z 
2022-09-27T16:40:08.7022110Z Running tests...
2022-09-27T16:40:08.7022607Z ----------------------------------------------------------------------
2022-09-27T16:40:08.9072012Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... [1664296808.906629] [28fd6574525b:106575:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:40:08.9073204Z [1664296808.906629] [28fd6574525b:106575:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:40:08.9106707Z [1664296808.910309] [28fd6574525b:106576:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:40:08.9107286Z [1664296808.910309] [28fd6574525b:106576:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:40:09.1862210Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:40:09.1884831Z 
2022-09-27T16:40:09.1885539Z Running tests...
2022-09-27T16:40:09.1886449Z ----------------------------------------------------------------------
2022-09-27T16:40:09.3028814Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:40:09 106575:106575 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:40:09.3030239Z STAGE:2022-09-27 16:40:09 106576:106576 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:40:10.3415116Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:10.3626505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106736
2022-09-27T16:40:10.3632005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106737
2022-09-27T16:40:10.6385344Z ok (7.500s)
2022-09-27T16:40:10.6385566Z 
2022-09-27T16:40:10.6386145Z ----------------------------------------------------------------------
2022-09-27T16:40:10.6386591Z Ran 1 test in 7.500s
2022-09-27T16:40:10.6386768Z 
2022-09-27T16:40:10.6386865Z OK
2022-09-27T16:40:10.6387005Z 
2022-09-27T16:40:10.6387124Z Generating XML reports...
2022-09-27T16:40:10.6421311Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164003.xml
2022-09-27T16:40:10.8441938Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:10.8647224Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106739
2022-09-27T16:40:10.8652348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106740
2022-09-27T16:40:12.0985600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:12.0986166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:12.0986770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:12.0987266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:12.1318337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:12.1318831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:12.1321305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:12.1321813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:12.6074870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:12.6075391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:12.6077626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:12.6078443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:12.6108717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:12.6109231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:12.6113508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:12.6114006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:12.6905843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:12.6906401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:12.6907170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:12.6907894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:12.6914838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:12.6915318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:12.8400763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:12.8401707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:12.8402907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:12.8403869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:13.1316944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:13.1333549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:13.1335009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:13.1418299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:13.1424844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:13.2350582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:13.3476845Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:13.3500598Z 
2022-09-27T16:40:13.3500997Z Running tests...
2022-09-27T16:40:13.3501903Z ----------------------------------------------------------------------
2022-09-27T16:40:13.3521100Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:40:13.3521810Z 
2022-09-27T16:40:13.3522343Z ----------------------------------------------------------------------
2022-09-27T16:40:13.3522975Z Ran 1 test in 0.002s
2022-09-27T16:40:13.3523255Z 
2022-09-27T16:40:13.3523452Z OK (skipped=1)
2022-09-27T16:40:13.3523735Z 
2022-09-27T16:40:13.3523962Z Generating XML reports...
2022-09-27T16:40:13.3559622Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164013.xml
2022-09-27T16:40:14.5329982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4f6qins
2022-09-27T16:40:14.5331333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4f6qins/_remote_module_non_scriptable.py
2022-09-27T16:40:14.5432196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gd6r3w1
2022-09-27T16:40:14.5434761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gd6r3w1/_remote_module_non_scriptable.py
2022-09-27T16:40:14.5675684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicmr9pcw
2022-09-27T16:40:14.5676962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicmr9pcw/_remote_module_non_scriptable.py
2022-09-27T16:40:14.5695738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1h15nt3
2022-09-27T16:40:14.5698825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1h15nt3/_remote_module_non_scriptable.py
2022-09-27T16:40:14.5908066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:14.5908594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:14.6014138Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpf4f6qins'>
2022-09-27T16:40:14.6014671Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:40:14.6015284Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpc1h15nt3'>
2022-09-27T16:40:14.6015734Z   _warnings.warn(warn_message, ResourceWarning)
2022-09-27T16:40:14.9704791Z STAGE:2022-09-27 16:40:14 106740:106740 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:40:14.9726844Z STAGE:2022-09-27 16:40:14 106739:106739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:40:14.9801501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:14.9802609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:14.9982716Z STAGE:2022-09-27 16:40:14 106740:106740 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:40:14.9996838Z STAGE:2022-09-27 16:40:14 106739:106739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:40:15.1067805Z STAGE:2022-09-27 16:40:15 106739:106739 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:40:15.1739801Z ok (6.471s)
2022-09-27T16:40:15.1740019Z 
2022-09-27T16:40:15.1740375Z ----------------------------------------------------------------------
2022-09-27T16:40:15.1740698Z Ran 1 test in 6.472s
2022-09-27T16:40:15.1740870Z 
2022-09-27T16:40:15.1740969Z OK
2022-09-27T16:40:15.1741107Z 
2022-09-27T16:40:15.1741246Z Generating XML reports...
2022-09-27T16:40:15.1778411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164008.xml
2022-09-27T16:40:15.3409751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:15.3410275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:15.3411098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:15.3411603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:15.8341454Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:15.8364167Z 
2022-09-27T16:40:15.8364492Z Running tests...
2022-09-27T16:40:15.8364942Z ----------------------------------------------------------------------
2022-09-27T16:40:15.8382437Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:40:15.8382822Z 
2022-09-27T16:40:15.8383093Z ----------------------------------------------------------------------
2022-09-27T16:40:15.8383452Z Ran 1 test in 0.002s
2022-09-27T16:40:15.8383620Z 
2022-09-27T16:40:15.8383731Z OK (skipped=1)
2022-09-27T16:40:15.8383888Z 
2022-09-27T16:40:15.8383999Z Generating XML reports...
2022-09-27T16:40:15.8417994Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164015.xml
2022-09-27T16:40:15.9413535Z STAGE:2022-09-27 16:40:15 106739:106739 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:40:16.5784051Z ok (7.390s)
2022-09-27T16:40:16.5784282Z 
2022-09-27T16:40:16.5784653Z ----------------------------------------------------------------------
2022-09-27T16:40:16.5785005Z Ran 1 test in 7.390s
2022-09-27T16:40:16.5785176Z 
2022-09-27T16:40:16.5785272Z OK
2022-09-27T16:40:16.5785413Z 
2022-09-27T16:40:16.5785550Z Generating XML reports...
2022-09-27T16:40:16.5842226Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164009.xml
2022-09-27T16:40:17.3473882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:17.3474395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:17.3475336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:17.3475823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:17.8334422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:17.8334940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:17.8336735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:17.8337231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:17.8416977Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:40:17.8441588Z 
2022-09-27T16:40:17.8441951Z Running tests...
2022-09-27T16:40:17.8442408Z ----------------------------------------------------------------------
2022-09-27T16:40:18.3292886Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:18.3313494Z 
2022-09-27T16:40:18.3313765Z Running tests...
2022-09-27T16:40:18.3314220Z ----------------------------------------------------------------------
2022-09-27T16:40:18.3335869Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:40:18.3336274Z 
2022-09-27T16:40:18.3336711Z ----------------------------------------------------------------------
2022-09-27T16:40:18.3337027Z Ran 1 test in 0.002s
2022-09-27T16:40:18.3337192Z 
2022-09-27T16:40:18.3337307Z OK (skipped=1)
2022-09-27T16:40:18.3337823Z 
2022-09-27T16:40:18.3337967Z Generating XML reports...
2022-09-27T16:40:18.3372087Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164018.xml
2022-09-27T16:40:18.6491032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:18.6491566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:18.6494034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:18.6494530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:19.1536604Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:40:19.1559078Z 
2022-09-27T16:40:19.1559512Z Running tests...
2022-09-27T16:40:19.1560009Z ----------------------------------------------------------------------
2022-09-27T16:40:19.4593357Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:19.4806964Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107106
2022-09-27T16:40:19.4814528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107107
2022-09-27T16:40:20.2797493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:20.2798490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:20.2799646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:20.2800944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:20.7748431Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:20.7749606Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:20.7773318Z 
2022-09-27T16:40:20.7773652Z Running tests...
2022-09-27T16:40:20.7774171Z ----------------------------------------------------------------------
2022-09-27T16:40:20.7798820Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:40:20.7799157Z 
2022-09-27T16:40:20.7799430Z ----------------------------------------------------------------------
2022-09-27T16:40:20.7799750Z Ran 1 test in 0.003s
2022-09-27T16:40:20.7799918Z 
2022-09-27T16:40:20.7800030Z OK (skipped=1)
2022-09-27T16:40:20.7800490Z 
2022-09-27T16:40:20.7800637Z Generating XML reports...
2022-09-27T16:40:20.7834449Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164020.xml
2022-09-27T16:40:20.7954820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107172
2022-09-27T16:40:20.7960754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107173
2022-09-27T16:40:21.1037950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:21.1038468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:21.1039543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:21.1040033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:21.1475917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:21.1476425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:21.1479495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:21.1479989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:21.7010777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:21.7011358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:21.7012129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:21.7012836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:21.7019437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:21.7019935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:21.7127938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6y_sre1
2022-09-27T16:40:21.7130302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppolragzm
2022-09-27T16:40:21.7130922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6y_sre1/_remote_module_non_scriptable.py
2022-09-27T16:40:21.7133741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppolragzm/_remote_module_non_scriptable.py
2022-09-27T16:40:21.7335365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:21.7335872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:22.3890493Z ok (4.545s)
2022-09-27T16:40:22.3890697Z 
2022-09-27T16:40:22.3891081Z ----------------------------------------------------------------------
2022-09-27T16:40:22.3891423Z Ran 1 test in 4.545s
2022-09-27T16:40:22.3891597Z 
2022-09-27T16:40:22.3891691Z OK
2022-09-27T16:40:22.3891834Z 
2022-09-27T16:40:22.3891951Z Generating XML reports...
2022-09-27T16:40:22.3927097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164017.xml
2022-09-27T16:40:22.4624609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:22.4625113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:22.4627666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:22.4628169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:22.5438469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:22.5438971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:22.5440933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:22.5441408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:22.8035918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:22.8036461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:22.8037613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:22.8038109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:22.9821056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:23.0562351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:23.0563144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:23.0631770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:23.0639142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:23.1575596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:23.2987036Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:23.3009272Z 
2022-09-27T16:40:23.3009737Z Running tests...
2022-09-27T16:40:23.3010253Z ----------------------------------------------------------------------
2022-09-27T16:40:23.3029677Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:40:23.3030230Z 
2022-09-27T16:40:23.3030537Z ----------------------------------------------------------------------
2022-09-27T16:40:23.3030863Z Ran 1 test in 0.002s
2022-09-27T16:40:23.3031029Z 
2022-09-27T16:40:23.3031150Z OK (skipped=1)
2022-09-27T16:40:23.3031317Z 
2022-09-27T16:40:23.3031446Z Generating XML reports...
2022-09-27T16:40:23.3066221Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164023.xml
2022-09-27T16:40:24.5276440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpuj0rt5
2022-09-27T16:40:24.5277322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpuj0rt5/_remote_module_non_scriptable.py
2022-09-27T16:40:24.5396452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgoqccuy
2022-09-27T16:40:24.5398632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgoqccuy/_remote_module_non_scriptable.py
2022-09-27T16:40:24.6027552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:24.6028322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:24.6031027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:24.6031507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:25.1175022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:40:25.1198877Z 
2022-09-27T16:40:25.1199103Z Running tests...
2022-09-27T16:40:25.1199546Z ----------------------------------------------------------------------
2022-09-27T16:40:25.2961328Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:25.2961930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:25.2962885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:25.2963359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:25.8053654Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:25.8075854Z 
2022-09-27T16:40:25.8076214Z Running tests...
2022-09-27T16:40:25.8076681Z ----------------------------------------------------------------------
2022-09-27T16:40:25.8096123Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:40:25.8096472Z 
2022-09-27T16:40:25.8096737Z ----------------------------------------------------------------------
2022-09-27T16:40:25.8097084Z Ran 1 test in 0.002s
2022-09-27T16:40:25.8097254Z 
2022-09-27T16:40:25.8097369Z OK (skipped=1)
2022-09-27T16:40:25.8097532Z 
2022-09-27T16:40:25.8099610Z Generating XML reports...
2022-09-27T16:40:25.8133409Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164025.xml
2022-09-27T16:40:26.3081201Z ok (7.152s)
2022-09-27T16:40:26.3081406Z 
2022-09-27T16:40:26.3082368Z ----------------------------------------------------------------------
2022-09-27T16:40:26.3082759Z Ran 1 test in 7.152s
2022-09-27T16:40:26.3082946Z 
2022-09-27T16:40:26.3083045Z OK
2022-09-27T16:40:26.3083198Z 
2022-09-27T16:40:26.3083338Z Generating XML reports...
2022-09-27T16:40:26.3118201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164019.xml
2022-09-27T16:40:26.8099125Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:26.8494022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107432
2022-09-27T16:40:26.8609331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107433
2022-09-27T16:40:27.9815055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:27.9815720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:27.9816789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:27.9817287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:28.4635595Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:28.4658292Z 
2022-09-27T16:40:28.4658590Z Running tests...
2022-09-27T16:40:28.4659044Z ----------------------------------------------------------------------
2022-09-27T16:40:28.4678084Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:40:28.4678756Z 
2022-09-27T16:40:28.4679056Z ----------------------------------------------------------------------
2022-09-27T16:40:28.4679379Z Ran 1 test in 0.002s
2022-09-27T16:40:28.4679693Z 
2022-09-27T16:40:28.4680061Z OK (skipped=1)
2022-09-27T16:40:28.4680808Z 
2022-09-27T16:40:28.4681081Z Generating XML reports...
2022-09-27T16:40:28.4714241Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164028.xml
2022-09-27T16:40:28.5469292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:28.5469942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:28.5471335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:28.5472158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:28.5661651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:28.5662243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:28.5663917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:28.5664763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:28.5784341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:28.5785113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:28.5787020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:28.5787829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:29.0499854Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:40:29.0520819Z 
2022-09-27T16:40:29.0521151Z Running tests...
2022-09-27T16:40:29.0522077Z ----------------------------------------------------------------------
2022-09-27T16:40:29.1020460Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:29.1021304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:29.1022067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:29.1022761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:29.1128826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:29.2033748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:30.4368862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:30.4369386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:30.4371290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:30.4371772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:30.5385584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h43txoo
2022-09-27T16:40:30.5387017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h43txoo/_remote_module_non_scriptable.py
2022-09-27T16:40:30.5834449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkxqfd61
2022-09-27T16:40:30.5836041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkxqfd61/_remote_module_non_scriptable.py
2022-09-27T16:40:30.7597384Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:30.7796463Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107576
2022-09-27T16:40:30.7801825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107577
2022-09-27T16:40:30.9603019Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:30.9625545Z 
2022-09-27T16:40:30.9625859Z Running tests...
2022-09-27T16:40:30.9626566Z ----------------------------------------------------------------------
2022-09-27T16:40:30.9650845Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:40:30.9651580Z 
2022-09-27T16:40:30.9652117Z ----------------------------------------------------------------------
2022-09-27T16:40:30.9652469Z Ran 1 test in 0.003s
2022-09-27T16:40:30.9652618Z 
2022-09-27T16:40:30.9652728Z OK (skipped=1)
2022-09-27T16:40:30.9652886Z 
2022-09-27T16:40:30.9653015Z Generating XML reports...
2022-09-27T16:40:30.9687975Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164030.xml
2022-09-27T16:40:31.0169645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:31.0170619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:31.6713980Z ok (6.551s)
2022-09-27T16:40:31.6714349Z 
2022-09-27T16:40:31.6715090Z ----------------------------------------------------------------------
2022-09-27T16:40:31.6715694Z Ran 1 test in 6.551s
2022-09-27T16:40:31.6715870Z 
2022-09-27T16:40:31.6715964Z OK
2022-09-27T16:40:31.6716107Z 
2022-09-27T16:40:31.6716249Z Generating XML reports...
2022-09-27T16:40:31.6752775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164025.xml
2022-09-27T16:40:32.4810258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:32.4810780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:32.4812810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:32.4813570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:32.4967641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:32.4968133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:32.4971383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:32.4971867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:33.0127231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:33.0127755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:33.0128956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:33.0129464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:33.0552887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:33.0741439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:33.0742529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:33.0756350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:33.0763722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:33.1756757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:33.5139781Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:33.5163064Z 
2022-09-27T16:40:33.5163499Z Running tests...
2022-09-27T16:40:33.5164011Z ----------------------------------------------------------------------
2022-09-27T16:40:33.5188093Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:40:33.5188651Z 
2022-09-27T16:40:33.5188929Z ----------------------------------------------------------------------
2022-09-27T16:40:33.5189403Z Ran 1 test in 0.003s
2022-09-27T16:40:33.5189695Z 
2022-09-27T16:40:33.5189834Z OK (skipped=1)
2022-09-27T16:40:33.5190003Z 
2022-09-27T16:40:33.5190133Z Generating XML reports...
2022-09-27T16:40:33.5225779Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164033.xml
2022-09-27T16:40:33.8036242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:33.8036747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:33.8038750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:33.8039233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:34.3232366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:40:34.3255386Z 
2022-09-27T16:40:34.3255810Z Running tests...
2022-09-27T16:40:34.3256317Z ----------------------------------------------------------------------
2022-09-27T16:40:34.3803042Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc__28b2_
2022-09-27T16:40:34.3804638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc__28b2_/_remote_module_non_scriptable.py
2022-09-27T16:40:34.5503666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ubuovvg
2022-09-27T16:40:34.5507064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ubuovvg/_remote_module_non_scriptable.py
2022-09-27T16:40:35.5042105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:35.5042936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:35.5043895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:35.5044634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:36.0069538Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:36.0265649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107763
2022-09-27T16:40:36.0272511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107764
2022-09-27T16:40:36.0463537Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:36.0489401Z 
2022-09-27T16:40:36.0489885Z Running tests...
2022-09-27T16:40:36.0490422Z ----------------------------------------------------------------------
2022-09-27T16:40:36.0512413Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:40:36.0513579Z 
2022-09-27T16:40:36.0514019Z ----------------------------------------------------------------------
2022-09-27T16:40:36.0514349Z Ran 1 test in 0.002s
2022-09-27T16:40:36.0514517Z 
2022-09-27T16:40:36.0514630Z OK (skipped=1)
2022-09-27T16:40:36.0514791Z 
2022-09-27T16:40:36.0514921Z Generating XML reports...
2022-09-27T16:40:36.0549646Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164036.xml
2022-09-27T16:40:36.0925736Z ok (7.040s)
2022-09-27T16:40:36.0926098Z 
2022-09-27T16:40:36.0926756Z ----------------------------------------------------------------------
2022-09-27T16:40:36.0927404Z Ran 1 test in 7.040s
2022-09-27T16:40:36.0927740Z 
2022-09-27T16:40:36.0927891Z OK
2022-09-27T16:40:36.0928149Z 
2022-09-27T16:40:36.0928389Z Generating XML reports...
2022-09-27T16:40:36.0965214Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164029.xml
2022-09-27T16:40:37.7409118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:37.7410087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:37.7411478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:37.7412180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:37.8623180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:37.8623897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:37.8626016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:37.8626794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:38.1905511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:38.1906121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:38.1908440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:38.1909219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:38.3298610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:38.3299164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:38.3302027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:38.3302774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:38.3816331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:38.3816984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:38.3817993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:38.3818716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:38.3925335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:38.4831734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:38.4939646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:40:38.4940497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:40:38.4941780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:40:38.4942567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:40:38.5148751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:40:38.5149279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:40:38.5149977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:40:38.5150669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:40:38.6820939Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:38.6843738Z 
2022-09-27T16:40:38.6843956Z Running tests...
2022-09-27T16:40:38.6844410Z ----------------------------------------------------------------------
2022-09-27T16:40:38.6873879Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-09-27T16:40:38.6874243Z 
2022-09-27T16:40:38.6874743Z ----------------------------------------------------------------------
2022-09-27T16:40:38.6875136Z Ran 1 test in 0.003s
2022-09-27T16:40:38.6875305Z 
2022-09-27T16:40:38.6875418Z OK (skipped=1)
2022-09-27T16:40:38.6875558Z 
2022-09-27T16:40:38.6875689Z Generating XML reports...
2022-09-27T16:40:38.6910002Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164038.xml
2022-09-27T16:40:38.8297192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:40:38.8321102Z 
2022-09-27T16:40:38.8321442Z Running tests...
2022-09-27T16:40:38.8321877Z ----------------------------------------------------------------------
2022-09-27T16:40:39.8955023Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ol8nya2
2022-09-27T16:40:39.8956417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ol8nya2/_remote_module_non_scriptable.py
2022-09-27T16:40:39.9241224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9786qen5
2022-09-27T16:40:39.9242334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9786qen5/_remote_module_non_scriptable.py
2022-09-27T16:40:40.4210743Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:40.4371965Z ok (6.111s)
2022-09-27T16:40:40.4372137Z 
2022-09-27T16:40:40.4372784Z ----------------------------------------------------------------------
2022-09-27T16:40:40.4373149Z Ran 1 test in 6.112s
2022-09-27T16:40:40.4373321Z 
2022-09-27T16:40:40.4373421Z OK
2022-09-27T16:40:40.4373569Z 
2022-09-27T16:40:40.4373704Z Generating XML reports...
2022-09-27T16:40:40.4406116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107951
2022-09-27T16:40:40.4410477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164034.xml
2022-09-27T16:40:40.4413237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107952
2022-09-27T16:40:40.6928764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:40.6929271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:40.6931907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:40.6932511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:41.3106403Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:41.3130658Z 
2022-09-27T16:40:41.3131171Z Running tests...
2022-09-27T16:40:41.3131753Z ----------------------------------------------------------------------
2022-09-27T16:40:41.3156168Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-09-27T16:40:41.3156651Z 
2022-09-27T16:40:41.3156942Z ----------------------------------------------------------------------
2022-09-27T16:40:41.3157287Z Ran 1 test in 0.003s
2022-09-27T16:40:41.3157589Z 
2022-09-27T16:40:41.3157762Z OK (skipped=1)
2022-09-27T16:40:41.3157997Z 
2022-09-27T16:40:41.3158131Z Generating XML reports...
2022-09-27T16:40:41.3193698Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164041.xml
2022-09-27T16:40:42.0944140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:42.0944648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:42.0945633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:42.0946248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:42.1989939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:42.1990427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:42.1991705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:42.1992269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:42.5717220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:42.5717730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:42.5720719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:42.5721778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:42.6578062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:42.7225398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:42.7226710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:42.7287113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:42.7293933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:42.8239480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:43.0742473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:40:43.0765927Z 
2022-09-27T16:40:43.0766331Z Running tests...
2022-09-27T16:40:43.0767203Z ----------------------------------------------------------------------
2022-09-27T16:40:43.3460649Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:43.3461787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:43.3462927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:43.3463837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:43.8697627Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:43.8721490Z 
2022-09-27T16:40:43.8721857Z Running tests...
2022-09-27T16:40:43.8722677Z ----------------------------------------------------------------------
2022-09-27T16:40:43.8744535Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:40:43.8745116Z 
2022-09-27T16:40:43.8745657Z ----------------------------------------------------------------------
2022-09-27T16:40:43.8746263Z Ran 1 test in 0.002s
2022-09-27T16:40:43.8746574Z 
2022-09-27T16:40:43.8746779Z OK (skipped=1)
2022-09-27T16:40:43.8747080Z 
2022-09-27T16:40:43.8747323Z Generating XML reports...
2022-09-27T16:40:43.8783887Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164043.xml
2022-09-27T16:40:44.0695379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14ag9rmf
2022-09-27T16:40:44.0696842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14ag9rmf/_remote_module_non_scriptable.py
2022-09-27T16:40:44.1910387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5mjbyju
2022-09-27T16:40:44.1911506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5mjbyju/_remote_module_non_scriptable.py
2022-09-27T16:40:44.6998271Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:44.7200858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108098
2022-09-27T16:40:44.7206508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108099
2022-09-27T16:40:45.7939474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:45.7940039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:40:46.0403637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:46.0404168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:46.0406971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:46.0407470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:46.4556103Z ok (7.623s)
2022-09-27T16:40:46.4556491Z 
2022-09-27T16:40:46.4557297Z ----------------------------------------------------------------------
2022-09-27T16:40:46.4557859Z Ran 1 test in 7.623s
2022-09-27T16:40:46.4558016Z 
2022-09-27T16:40:46.4558109Z OK
2022-09-27T16:40:46.4558249Z 
2022-09-27T16:40:46.4558386Z Generating XML reports...
2022-09-27T16:40:46.4578083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:46.4579118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:46.4582586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:46.4583401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:46.4593866Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164038.xml
2022-09-27T16:40:46.4910081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:46.4910793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:46.4912041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:46.4912946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:46.5301529Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:46.5325088Z 
2022-09-27T16:40:46.5325283Z Running tests...
2022-09-27T16:40:46.5326210Z ----------------------------------------------------------------------
2022-09-27T16:40:46.5350797Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:40:46.5351607Z 
2022-09-27T16:40:46.5352262Z ----------------------------------------------------------------------
2022-09-27T16:40:46.5353008Z Ran 1 test in 0.003s
2022-09-27T16:40:46.5353301Z 
2022-09-27T16:40:46.5353428Z OK (skipped=1)
2022-09-27T16:40:46.5353570Z 
2022-09-27T16:40:46.5353698Z Generating XML reports...
2022-09-27T16:40:46.5388624Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164046.xml
2022-09-27T16:40:47.0213743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:47.0214962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:47.0215836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:47.0216553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:47.0321611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:47.1227729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:47.1434602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:40:47.1435746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:40:47.1436625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:40:47.1437314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:40:47.1642907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:40:47.1644037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:40:47.1644785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:40:47.1645464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:40:48.5636131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:48.5636633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:48.5639305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:48.5639832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:48.5940376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq2oociz
2022-09-27T16:40:48.5941549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq2oociz/_remote_module_non_scriptable.py
2022-09-27T16:40:48.5949651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjb5ls0d5
2022-09-27T16:40:48.5953075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjb5ls0d5/_remote_module_non_scriptable.py
2022-09-27T16:40:48.6629057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:48.6629580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:48.6631286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:48.6631784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:49.0620754Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:49.0643765Z 
2022-09-27T16:40:49.0643913Z Running tests...
2022-09-27T16:40:49.0645228Z ----------------------------------------------------------------------
2022-09-27T16:40:49.0665783Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:40:49.0666165Z 
2022-09-27T16:40:49.0666471Z ----------------------------------------------------------------------
2022-09-27T16:40:49.0666807Z Ran 1 test in 0.002s
2022-09-27T16:40:49.0666980Z 
2022-09-27T16:40:49.0667094Z OK (skipped=1)
2022-09-27T16:40:49.0667260Z 
2022-09-27T16:40:49.0667408Z Generating XML reports...
2022-09-27T16:40:49.0701904Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164049.xml
2022-09-27T16:40:49.1592280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:40:49.1616291Z 
2022-09-27T16:40:49.1616501Z Running tests...
2022-09-27T16:40:49.1616936Z ----------------------------------------------------------------------
2022-09-27T16:40:50.8029650Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:50.8196622Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s)
2022-09-27T16:40:50.8197219Z 
2022-09-27T16:40:50.8197531Z ----------------------------------------------------------------------
2022-09-27T16:40:50.8197875Z Ran 1 test in 1.658s
2022-09-27T16:40:50.8198045Z 
2022-09-27T16:40:50.8198158Z OK (skipped=1)
2022-09-27T16:40:50.8198318Z 
2022-09-27T16:40:50.8198436Z Generating XML reports...
2022-09-27T16:40:50.8232400Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164049.xml
2022-09-27T16:40:51.0140387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:51.0140944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:51.0143302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:51.0143805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:51.5143707Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:51.5168036Z 
2022-09-27T16:40:51.5168613Z Running tests...
2022-09-27T16:40:51.5169555Z ----------------------------------------------------------------------
2022-09-27T16:40:52.8918422Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:52.8919636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:52.8920784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:52.8921276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:53.1204089Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:53.1415069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108362
2022-09-27T16:40:53.1422688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108363
2022-09-27T16:40:53.4012912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:40:53.4036434Z 
2022-09-27T16:40:53.4036846Z Running tests...
2022-09-27T16:40:53.4037637Z ----------------------------------------------------------------------
2022-09-27T16:40:54.7526904Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:54.7527525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:54.7528352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:54.7528844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:54.8131026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:54.8131554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:54.8133534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:54.8134036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:55.0221320Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:55.0389261Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2022-09-27T16:40:55.0389909Z 
2022-09-27T16:40:55.0390198Z ----------------------------------------------------------------------
2022-09-27T16:40:55.0390517Z Ran 1 test in 1.635s
2022-09-27T16:40:55.0390684Z 
2022-09-27T16:40:55.0390797Z OK (skipped=1)
2022-09-27T16:40:55.0390954Z 
2022-09-27T16:40:55.0391096Z Generating XML reports...
2022-09-27T16:40:55.0424512Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164053.xml
2022-09-27T16:40:55.2806827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:40:55.3242895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:40:55.3243732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:55.3313855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:40:55.3320981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:40:55.4255607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:40:55.7486550Z ok (4.232s)
2022-09-27T16:40:55.7486769Z 
2022-09-27T16:40:55.7487157Z ----------------------------------------------------------------------
2022-09-27T16:40:55.7487484Z Ran 1 test in 4.232s
2022-09-27T16:40:55.7487667Z 
2022-09-27T16:40:55.7487763Z OK
2022-09-27T16:40:55.7487904Z 
2022-09-27T16:40:55.7488039Z Generating XML reports...
2022-09-27T16:40:55.7524888Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164051.xml
2022-09-27T16:40:57.1684539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:57.1685058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:57.1687363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:57.1687844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:57.6614598Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:40:57.6637239Z 
2022-09-27T16:40:57.6637774Z Running tests...
2022-09-27T16:40:57.6638714Z ----------------------------------------------------------------------
2022-09-27T16:40:57.7637865Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:40:57.7638517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:40:57.7641052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:40:57.7641929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:40:58.2463168Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:40:58.2486171Z 
2022-09-27T16:40:58.2486489Z Running tests...
2022-09-27T16:40:58.2487183Z ----------------------------------------------------------------------
2022-09-27T16:40:59.1490176Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... ok (16.072s)
2022-09-27T16:40:59.1490981Z 
2022-09-27T16:40:59.1491461Z ----------------------------------------------------------------------
2022-09-27T16:40:59.1491810Z Ran 1 test in 16.072s
2022-09-27T16:40:59.1491960Z 
2022-09-27T16:40:59.1492058Z OK
2022-09-27T16:40:59.1492194Z 
2022-09-27T16:40:59.1492325Z Generating XML reports...
2022-09-27T16:40:59.1526616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164043.xml
2022-09-27T16:40:59.3024907Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:59.3230616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108500
2022-09-27T16:40:59.3235717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108501
2022-09-27T16:40:59.9047852Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:40:59.9242844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108504
2022-09-27T16:40:59.9248956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108505
2022-09-27T16:41:01.1446566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:01.1447112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:01.1448390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:01.1448867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:01.1588104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:01.1588590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:01.1591882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:01.1592395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:01.3992350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:01.3992874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:01.3995404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:01.3995904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:01.6744016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:01.6751860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:01.6753132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:01.6844867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:01.6851407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:01.7241491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:01.7242015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:01.7244496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:01.7244998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:01.7767334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:01.8003317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:01.8003801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:01.8006144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:01.8006638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:01.8987920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:41:01.9011999Z 
2022-09-27T16:41:01.9012268Z Running tests...
2022-09-27T16:41:01.9012688Z ----------------------------------------------------------------------
2022-09-27T16:41:02.2472332Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:02.3443001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:02.3443845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:02.3488768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:02.3497129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:02.4456217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:02.9322902Z ok (4.683s)
2022-09-27T16:41:02.9323106Z 
2022-09-27T16:41:02.9323495Z ----------------------------------------------------------------------
2022-09-27T16:41:02.9323836Z Ran 1 test in 4.683s
2022-09-27T16:41:02.9324004Z 
2022-09-27T16:41:02.9324106Z OK
2022-09-27T16:41:02.9324245Z 
2022-09-27T16:41:02.9324364Z Generating XML reports...
2022-09-27T16:41:02.9359671Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164058.xml
2022-09-27T16:41:03.0723753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvyhyhwr
2022-09-27T16:41:03.0724754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvyhyhwr/_remote_module_non_scriptable.py
2022-09-27T16:41:03.1411404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2ifj8nc
2022-09-27T16:41:03.1412376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2ifj8nc/_remote_module_non_scriptable.py
2022-09-27T16:41:03.5659620Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:03.5862069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108682
2022-09-27T16:41:03.5867732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108683
2022-09-27T16:41:05.1762045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:05.1762568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:05.1765060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:05.1765885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:05.2194334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:05.2194845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:05.2197009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:05.2197498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:05.2752734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:05.2753243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:05.2755719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:05.2756223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:05.4368516Z ok (7.773s)
2022-09-27T16:41:05.4368730Z 
2022-09-27T16:41:05.4369105Z ----------------------------------------------------------------------
2022-09-27T16:41:05.4369443Z Ran 1 test in 7.773s
2022-09-27T16:41:05.4369613Z 
2022-09-27T16:41:05.4369713Z OK
2022-09-27T16:41:05.4369829Z 
2022-09-27T16:41:05.4369970Z Generating XML reports...
2022-09-27T16:41:05.4405190Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164057.xml
2022-09-27T16:41:05.6976882Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:41:05.6998619Z 
2022-09-27T16:41:05.6998892Z Running tests...
2022-09-27T16:41:05.6999310Z ----------------------------------------------------------------------
2022-09-27T16:41:05.7019434Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:41:05.7019787Z 
2022-09-27T16:41:05.7020082Z ----------------------------------------------------------------------
2022-09-27T16:41:05.7020412Z Ran 1 test in 0.002s
2022-09-27T16:41:05.7020562Z 
2022-09-27T16:41:05.7020673Z OK (skipped=1)
2022-09-27T16:41:05.7020833Z 
2022-09-27T16:41:05.7020956Z Generating XML reports...
2022-09-27T16:41:05.7055420Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164105.xml
2022-09-27T16:41:05.8100156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:05.8101534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:05.8102368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:05.8103088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:05.8210899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:05.9115209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:07.2896440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8aekntyq
2022-09-27T16:41:07.2897360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8aekntyq/_remote_module_non_scriptable.py
2022-09-27T16:41:07.3303333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrad9q13
2022-09-27T16:41:07.3304604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrad9q13/_remote_module_non_scriptable.py
2022-09-27T16:41:07.5895200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:07.5896108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:07.5897380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:07.5897884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:07.6542423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:07.6542925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:07.6544475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:07.6545130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:08.1115630Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:41:08.1138983Z 
2022-09-27T16:41:08.1139356Z Running tests...
2022-09-27T16:41:08.1140050Z ----------------------------------------------------------------------
2022-09-27T16:41:08.1501804Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:41:08.1525181Z 
2022-09-27T16:41:08.1525420Z Running tests...
2022-09-27T16:41:08.1526045Z ----------------------------------------------------------------------
2022-09-27T16:41:08.1545547Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:41:08.1545948Z 
2022-09-27T16:41:08.1546288Z ----------------------------------------------------------------------
2022-09-27T16:41:08.1546628Z Ran 1 test in 0.002s
2022-09-27T16:41:08.1546814Z 
2022-09-27T16:41:08.1546927Z OK (skipped=1)
2022-09-27T16:41:08.1547070Z 
2022-09-27T16:41:08.1547198Z Generating XML reports...
2022-09-27T16:41:08.1582384Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164108.xml
2022-09-27T16:41:08.7982941Z ok (6.897s)
2022-09-27T16:41:08.7983311Z 
2022-09-27T16:41:08.7983708Z ----------------------------------------------------------------------
2022-09-27T16:41:08.7984053Z Ran 1 test in 6.897s
2022-09-27T16:41:08.7984221Z 
2022-09-27T16:41:08.7984297Z OK
2022-09-27T16:41:08.7984433Z 
2022-09-27T16:41:08.7984568Z Generating XML reports...
2022-09-27T16:41:08.8020979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164101.xml
2022-09-27T16:41:09.7752264Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:09.7950760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108904
2022-09-27T16:41:09.7956922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108905
2022-09-27T16:41:10.1861868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:10.1862404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:10.1865077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:10.1865573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:10.7962379Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:41:10.7983845Z 
2022-09-27T16:41:10.7984278Z Running tests...
2022-09-27T16:41:10.7984760Z ----------------------------------------------------------------------
2022-09-27T16:41:10.9971255Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:10.9971864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:10.9973100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:10.9973589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:11.4725517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:11.4726503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:11.4728330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:11.4729303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:11.5037440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:41:11.5060426Z 
2022-09-27T16:41:11.5060878Z Running tests...
2022-09-27T16:41:11.5061372Z ----------------------------------------------------------------------
2022-09-27T16:41:11.5297884Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:11.5298486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:11.5301546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:11.5302047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:11.9909018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:12.0595424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:12.0596256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:12.0618557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:12.0625509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:12.0629567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:41:12.1609235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:12.1613037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:41:12.1613790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:41:12.1645028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:41:12.4012848Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:12.4185678Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2022-09-27T16:41:12.4186291Z 
2022-09-27T16:41:12.4186567Z ----------------------------------------------------------------------
2022-09-27T16:41:12.4186913Z Ran 1 test in 1.620s
2022-09-27T16:41:12.4187079Z 
2022-09-27T16:41:12.4187195Z OK (skipped=1)
2022-09-27T16:41:12.4187340Z 
2022-09-27T16:41:12.4187467Z Generating XML reports...
2022-09-27T16:41:12.4221236Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164110.xml
2022-09-27T16:41:12.7033499Z ok (4.589s)
2022-09-27T16:41:12.7033780Z 
2022-09-27T16:41:12.7034186Z ----------------------------------------------------------------------
2022-09-27T16:41:12.7034531Z Ran 1 test in 4.589s
2022-09-27T16:41:12.7035000Z 
2022-09-27T16:41:12.7035095Z OK
2022-09-27T16:41:12.7035213Z 
2022-09-27T16:41:12.7035354Z Generating XML reports...
2022-09-27T16:41:12.7071166Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164108.xml
2022-09-27T16:41:13.0963129Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:13.1160038Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109017
2022-09-27T16:41:13.1165899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109018
2022-09-27T16:41:14.6928619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:14.6929150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:14.6930305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:14.6930782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:14.9050144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:14.9050674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:14.9051849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:14.9052344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:14.9140859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:14.9141330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:14.9145436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:14.9145929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:14.9919113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:14.9919611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:14.9921139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:14.9921637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:15.1827744Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:41:15.1849861Z 
2022-09-27T16:41:15.1850098Z Running tests...
2022-09-27T16:41:15.1850524Z ----------------------------------------------------------------------
2022-09-27T16:41:15.4397591Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:15.4398293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:15.4399109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:15.4399803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:15.4405446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:15.4405950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:15.4943071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:41:15.4966413Z 
2022-09-27T16:41:15.4966667Z Running tests...
2022-09-27T16:41:15.4967110Z ----------------------------------------------------------------------
2022-09-27T16:41:16.8276328Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4bfzflm
2022-09-27T16:41:16.8277297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4bfzflm/_remote_module_non_scriptable.py
2022-09-27T16:41:16.8436392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn39_fnf4
2022-09-27T16:41:16.8438412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn39_fnf4/_remote_module_non_scriptable.py
2022-09-27T16:41:16.8815640Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:16.9017317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109161
2022-09-27T16:41:16.9021858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109162
2022-09-27T16:41:17.1645486Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:17.1970868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109164
2022-09-27T16:41:17.1976371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109165
2022-09-27T16:41:18.2284542Z ok (6.722s)
2022-09-27T16:41:18.2284757Z 
2022-09-27T16:41:18.2285159Z ----------------------------------------------------------------------
2022-09-27T16:41:18.2285505Z Ran 1 test in 6.722s
2022-09-27T16:41:18.2285675Z 
2022-09-27T16:41:18.2285772Z OK
2022-09-27T16:41:18.2285912Z 
2022-09-27T16:41:18.2286032Z Generating XML reports...
2022-09-27T16:41:18.2338156Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164111.xml
2022-09-27T16:41:18.5989329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:18.5989839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:18.5990596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:18.5991087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:18.6548162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:18.6548793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:18.6550142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:18.6550634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:18.9647460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:18.9647970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:18.9649376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:18.9649894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:18.9680323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:18.9681023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:18.9684659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:18.9685151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:19.1083760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:19.1753500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:19.1754316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:19.1792671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:19.1799480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:19.2767706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:19.5413065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:19.5428613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:19.5429431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:19.5515132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:19.5521688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:19.5525158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:41:19.6442524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:19.6445501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:41:19.6446275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:41:19.6541152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:41:20.2048866Z ok (4.708s)
2022-09-27T16:41:20.2049066Z 
2022-09-27T16:41:20.2049478Z ----------------------------------------------------------------------
2022-09-27T16:41:20.2049831Z Ran 1 test in 4.708s
2022-09-27T16:41:20.2050001Z 
2022-09-27T16:41:20.2050118Z OK
2022-09-27T16:41:20.2050255Z 
2022-09-27T16:41:20.2050373Z Generating XML reports...
2022-09-27T16:41:20.2086963Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164115.xml
2022-09-27T16:41:20.4458396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:20.4458896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:20.4459936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:20.4460428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:20.5844308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5et_fcxc
2022-09-27T16:41:20.5845681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5et_fcxc/_remote_module_non_scriptable.py
2022-09-27T16:41:20.6444041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cugxkqa
2022-09-27T16:41:20.6445456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cugxkqa/_remote_module_non_scriptable.py
2022-09-27T16:41:20.6713444Z [1664296880.670957] [28fd6574525b:109161:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:41:20.6713938Z [1664296880.670957] [28fd6574525b:109161:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:41:20.6825875Z [1664296880.682219] [28fd6574525b:109162:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:41:20.6826374Z [1664296880.682219] [28fd6574525b:109162:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:41:20.9504579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:41:20.9527565Z 
2022-09-27T16:41:20.9528022Z Running tests...
2022-09-27T16:41:20.9528521Z ----------------------------------------------------------------------
2022-09-27T16:41:22.2144915Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... ok (7.029s)
2022-09-27T16:41:22.2145691Z 
2022-09-27T16:41:22.2146088Z ----------------------------------------------------------------------
2022-09-27T16:41:22.2146413Z Ran 1 test in 7.029s
2022-09-27T16:41:22.2146580Z 
2022-09-27T16:41:22.2146672Z OK
2022-09-27T16:41:22.2146809Z 
2022-09-27T16:41:22.2146939Z Generating XML reports...
2022-09-27T16:41:22.2181662Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164115.xml
2022-09-27T16:41:22.3597729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:22.3598237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:22.3599777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:22.3600589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:22.6745894Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:22.6942946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109394
2022-09-27T16:41:22.6948968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109395
2022-09-27T16:41:22.8751797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:41:22.8774600Z 
2022-09-27T16:41:22.8774937Z Running tests...
2022-09-27T16:41:22.8775383Z ----------------------------------------------------------------------
2022-09-27T16:41:24.3086391Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:24.3087034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:24.3088196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:24.3088701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:24.3225897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:24.3226380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:24.3229867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:24.3230355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:24.3234881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:24.3235618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:24.3239176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:24.3239669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:24.5137914Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:24.5330028Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109494
2022-09-27T16:41:24.5336458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109495
2022-09-27T16:41:24.8362560Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:41:24.8385416Z 
2022-09-27T16:41:24.8385875Z Running tests...
2022-09-27T16:41:24.8386329Z ----------------------------------------------------------------------
2022-09-27T16:41:24.8744682Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:24.8745310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:24.8746021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:24.8746941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:24.8754204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:24.8754699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:26.1499925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:26.1500959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:26.1502135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:26.1503025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:26.1504159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:26.1505099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:26.1506305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:26.1507247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:26.3085290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgu0lgnhh
2022-09-27T16:41:26.3086469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgu0lgnhh/_remote_module_non_scriptable.py
2022-09-27T16:41:26.3460879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtn92wpu
2022-09-27T16:41:26.3462053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtn92wpu/_remote_module_non_scriptable.py
2022-09-27T16:41:26.5909789Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:26.6118591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109572
2022-09-27T16:41:26.6123767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109573
2022-09-27T16:41:26.7035451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:26.7074235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:26.7075746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:26.7137146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:26.7143630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:26.8091582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:27.5077949Z ok (6.555s)
2022-09-27T16:41:27.5078187Z 
2022-09-27T16:41:27.5078594Z ----------------------------------------------------------------------
2022-09-27T16:41:27.5078919Z Ran 1 test in 6.555s
2022-09-27T16:41:27.5079092Z 
2022-09-27T16:41:27.5079187Z OK
2022-09-27T16:41:27.5079325Z 
2022-09-27T16:41:27.5079468Z Generating XML reports...
2022-09-27T16:41:27.5138667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164120.xml
2022-09-27T16:41:28.1377598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaasq_lgl
2022-09-27T16:41:28.1378751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaasq_lgl/_remote_module_non_scriptable.py
2022-09-27T16:41:28.2516283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:28.2516783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:28.2519521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:28.2520040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:28.2740546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_uv8zdg
2022-09-27T16:41:28.2742720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_uv8zdg/_remote_module_non_scriptable.py
2022-09-27T16:41:28.2857480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:28.2858004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:28.2861508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:28.2862015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:28.8091259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:28.8092427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:28.8167114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:28.8188796Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:41:28.8195006Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:41:28.8301753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:28.8302569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:28.8370388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:28.8376627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:28.9316847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:29.4446750Z ok (6.567s)
2022-09-27T16:41:29.4446958Z 
2022-09-27T16:41:29.4447345Z ----------------------------------------------------------------------
2022-09-27T16:41:29.4447684Z Ran 1 test in 6.567s
2022-09-27T16:41:29.4447852Z 
2022-09-27T16:41:29.4447948Z OK
2022-09-27T16:41:29.4448083Z 
2022-09-27T16:41:29.4448218Z Generating XML reports...
2022-09-27T16:41:29.4483203Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164122.xml
2022-09-27T16:41:29.7601177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:29.7601717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:29.7602854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:29.7603661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:30.2844267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:41:30.2867824Z 
2022-09-27T16:41:30.2868045Z Running tests...
2022-09-27T16:41:30.2868774Z ----------------------------------------------------------------------
2022-09-27T16:41:31.4938859Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... [1664296891.493259] [28fd6574525b:109572:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:41:31.4939505Z [1664296891.493259] [28fd6574525b:109572:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:41:31.5002598Z [1664296891.499881] [28fd6574525b:109573:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:41:31.5003632Z [1664296891.499881] [28fd6574525b:109573:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:41:31.6505449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:31.6506032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:31.6507750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:31.6508533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:31.9680688Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:31.9880138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109739
2022-09-27T16:41:31.9885546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109740
2022-09-27T16:41:32.1802167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:41:32.1825090Z 
2022-09-27T16:41:32.1825648Z Running tests...
2022-09-27T16:41:32.1826563Z ----------------------------------------------------------------------
2022-09-27T16:41:32.5251490Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... ok (7.686s)
2022-09-27T16:41:32.5251996Z 
2022-09-27T16:41:32.5252534Z ----------------------------------------------------------------------
2022-09-27T16:41:32.5252884Z Ran 1 test in 7.686s
2022-09-27T16:41:32.5253051Z 
2022-09-27T16:41:32.5253145Z OK
2022-09-27T16:41:32.5253282Z 
2022-09-27T16:41:32.5253392Z Generating XML reports...
2022-09-27T16:41:32.5294421Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164124.xml
2022-09-27T16:41:33.6530059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:33.6530615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:33.6531227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:33.6531704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:33.6641372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:33.6642324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:33.6648895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:33.6649894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:33.9798931Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:33.9989137Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109839
2022-09-27T16:41:33.9995091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109840
2022-09-27T16:41:34.2594600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:34.2595349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:34.2596140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:34.2596845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:34.2602843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:34.2603738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:34.7155662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:34.7156762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:34.7159423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:34.7161082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:35.2319506Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:41:35.2342538Z 
2022-09-27T16:41:35.2342694Z Running tests...
2022-09-27T16:41:35.2343147Z ----------------------------------------------------------------------
2022-09-27T16:41:35.6875229Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbf_adca
2022-09-27T16:41:35.6876168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbf_adca/_remote_module_non_scriptable.py
2022-09-27T16:41:35.6883396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:35.6884190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:35.6887618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:35.6888381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:35.6961950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpll7s8wnt
2022-09-27T16:41:35.6965141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpll7s8wnt/_remote_module_non_scriptable.py
2022-09-27T16:41:35.7197289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:35.7197787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:35.7200232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:35.7200991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:36.1290438Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:41:36.1382935Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:41:36.1487300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:36.1487807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:36.2117777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:36.2656545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:36.2657375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:36.2729258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:36.2735527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:36.3671537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:36.6990445Z ok (6.412s)
2022-09-27T16:41:36.6990669Z 
2022-09-27T16:41:36.6991396Z ----------------------------------------------------------------------
2022-09-27T16:41:36.6991764Z Ran 1 test in 6.412s
2022-09-27T16:41:36.6991938Z 
2022-09-27T16:41:36.6992036Z OK
2022-09-27T16:41:36.6992180Z 
2022-09-27T16:41:36.6992323Z Generating XML reports...
2022-09-27T16:41:36.7029358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164130.xml
2022-09-27T16:41:36.9311408Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:36.9516749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109930
2022-09-27T16:41:36.9522161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109931
2022-09-27T16:41:37.6935011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzubok_76
2022-09-27T16:41:37.6937169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzubok_76/_remote_module_non_scriptable.py
2022-09-27T16:41:37.8237723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdj1d161
2022-09-27T16:41:37.8239189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdj1d161/_remote_module_non_scriptable.py
2022-09-27T16:41:38.4513914Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:41:38.4777189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:38.4777712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:38.6749613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:38.6750136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:38.6751739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:38.6752223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:38.7258453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:38.7258991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:38.7260760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:38.7261651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:39.0599241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:39.0599756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:39.0602328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:39.0602806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:39.1105900Z ok (6.928s)
2022-09-27T16:41:39.1106092Z 
2022-09-27T16:41:39.1107186Z ----------------------------------------------------------------------
2022-09-27T16:41:39.1107826Z Ran 1 test in 6.928s
2022-09-27T16:41:39.1108004Z 
2022-09-27T16:41:39.1108101Z OK
2022-09-27T16:41:39.1108238Z 
2022-09-27T16:41:39.1108374Z Generating XML reports...
2022-09-27T16:41:39.1143345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164132.xml
2022-09-27T16:41:39.1857397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:39.2469015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:39.2469812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:39.2567591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:39.2574475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:39.3482716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:39.5790931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:41:39.5814162Z 
2022-09-27T16:41:39.5814563Z Running tests...
2022-09-27T16:41:39.5815012Z ----------------------------------------------------------------------
2022-09-27T16:41:41.2289791Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:41.2494854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110079
2022-09-27T16:41:41.2500131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110080
2022-09-27T16:41:41.2855574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:41.2856547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:41.2858056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:41.2859012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:41.7911387Z [1664296901.790334] [28fd6574525b:109931:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:41:41.7912558Z [1664296901.790334] [28fd6574525b:109931:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:41:41.7921542Z [1664296901.791675] [28fd6574525b:109930:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:41:41.7922700Z [1664296901.791675] [28fd6574525b:109930:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:41:41.7936236Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:41:41.7960163Z 
2022-09-27T16:41:41.7960614Z Running tests...
2022-09-27T16:41:41.7961923Z ----------------------------------------------------------------------
2022-09-27T16:41:42.8649501Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... ok (7.630s)
2022-09-27T16:41:42.8650239Z 
2022-09-27T16:41:42.8650634Z ----------------------------------------------------------------------
2022-09-27T16:41:42.8650974Z Ran 1 test in 7.630s
2022-09-27T16:41:42.8651122Z 
2022-09-27T16:41:42.8651216Z OK
2022-09-27T16:41:42.8651352Z 
2022-09-27T16:41:42.8651483Z Generating XML reports...
2022-09-27T16:41:42.8686223Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164135.xml
2022-09-27T16:41:42.9082365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:42.9082854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:42.9084865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:42.9085356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:42.9608813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:42.9609297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:42.9612454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:42.9612940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:43.4681760Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:43.4882357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110160
2022-09-27T16:41:43.4889004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110161
2022-09-27T16:41:43.4995490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:43.4996012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:43.4996815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:43.4997507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:43.5105487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:43.6012210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:45.0149431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6oiq0hus
2022-09-27T16:41:45.0150623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6oiq0hus/_remote_module_non_scriptable.py
2022-09-27T16:41:45.0185627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8k1s2zzh
2022-09-27T16:41:45.0187081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8k1s2zzh/_remote_module_non_scriptable.py
2022-09-27T16:41:45.0375850Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:41:45.0377332Z   warnings.warn(
2022-09-27T16:41:45.0379457Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:41:45.0380911Z   warnings.warn(
2022-09-27T16:41:45.0627119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:45.0628540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:45.0629714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:45.0630678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:45.2286387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:45.2287356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:45.2289060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:45.2290023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:45.2360962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:45.2361868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:45.2364246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:45.2365119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:45.5694545Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:41:45.5717423Z 
2022-09-27T16:41:45.5718016Z Running tests...
2022-09-27T16:41:45.5718547Z ----------------------------------------------------------------------
2022-09-27T16:41:45.7629528Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:45.7708006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:45.7709619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:45.7730292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:45.7737324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:45.8722390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:46.0607012Z ok (6.479s)
2022-09-27T16:41:46.0607237Z 
2022-09-27T16:41:46.0607629Z ----------------------------------------------------------------------
2022-09-27T16:41:46.0607967Z Ran 1 test in 6.479s
2022-09-27T16:41:46.0608138Z 
2022-09-27T16:41:46.0609758Z OK
2022-09-27T16:41:46.0611101Z 
2022-09-27T16:41:46.0611262Z Generating XML reports...
2022-09-27T16:41:46.0644535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164139.xml
2022-09-27T16:41:46.2960646Z ok (4.500s)
2022-09-27T16:41:46.2961112Z 
2022-09-27T16:41:46.2961500Z ----------------------------------------------------------------------
2022-09-27T16:41:46.2961840Z Ran 1 test in 4.500s
2022-09-27T16:41:46.2962009Z 
2022-09-27T16:41:46.2962092Z OK
2022-09-27T16:41:46.2962229Z 
2022-09-27T16:41:46.2962362Z Generating XML reports...
2022-09-27T16:41:46.2997788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164141.xml
2022-09-27T16:41:47.3219128Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:47.3444397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110311
2022-09-27T16:41:47.3450525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110312
2022-09-27T16:41:48.3975947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:48.3976458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:48.3978330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:48.3978836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:48.4487713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:48.4488199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:48.4491161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:48.4491741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:48.9088224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:41:48.9111177Z 
2022-09-27T16:41:48.9111631Z Running tests...
2022-09-27T16:41:48.9112452Z ----------------------------------------------------------------------
2022-09-27T16:41:48.9406755Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:41:48.9428921Z 
2022-09-27T16:41:48.9429422Z Running tests...
2022-09-27T16:41:48.9430057Z ----------------------------------------------------------------------
2022-09-27T16:41:48.9447688Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:41:48.9448215Z 
2022-09-27T16:41:48.9448823Z ----------------------------------------------------------------------
2022-09-27T16:41:48.9449474Z Ran 1 test in 0.002s
2022-09-27T16:41:48.9449661Z 
2022-09-27T16:41:48.9449771Z OK (skipped=1)
2022-09-27T16:41:48.9449938Z 
2022-09-27T16:41:48.9450064Z Generating XML reports...
2022-09-27T16:41:48.9483466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164148.xml
2022-09-27T16:41:49.0569746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:49.0570417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:49.0571819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:49.0572630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:49.1061919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:49.1062484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:49.1064608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:49.1065201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:49.5835733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:49.6345793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:49.6346985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:49.6443306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:49.6449915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:49.7360681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:50.5661272Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:50.5873962Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110448
2022-09-27T16:41:50.5878793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110449
2022-09-27T16:41:50.9385473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:50.9386165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:50.9387455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:50.9388014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:51.0117111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29pr5ls5
2022-09-27T16:41:51.0117961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29pr5ls5/_remote_module_non_scriptable.py
2022-09-27T16:41:51.0960637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjo0kpm8
2022-09-27T16:41:51.0961924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjo0kpm8/_remote_module_non_scriptable.py
2022-09-27T16:41:51.1305334Z [1664296911.130041] [28fd6574525b:110311:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:41:51.1306305Z [1664296911.130041] [28fd6574525b:110311:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:41:51.1339007Z [1664296911.133580] [28fd6574525b:110312:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:41:51.1339997Z [1664296911.133580] [28fd6574525b:110312:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:41:51.4958156Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:41:51.4980619Z 
2022-09-27T16:41:51.4980942Z Running tests...
2022-09-27T16:41:51.4981402Z ----------------------------------------------------------------------
2022-09-27T16:41:51.5005716Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:41:51.5006046Z 
2022-09-27T16:41:51.5006329Z ----------------------------------------------------------------------
2022-09-27T16:41:51.5006641Z Ran 1 test in 0.003s
2022-09-27T16:41:51.5006807Z 
2022-09-27T16:41:51.5006918Z OK (skipped=1)
2022-09-27T16:41:51.5007077Z 
2022-09-27T16:41:51.5007212Z Generating XML reports...
2022-09-27T16:41:51.5041276Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164151.xml
2022-09-27T16:41:51.9939772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:51.9940318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:51.9956718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:51.9957229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:52.2570465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:52.2571424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:52.2573111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:52.2573973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:52.2846974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:52.2847505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:52.2848323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:52.2848804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:52.6572818Z ok (7.085s)
2022-09-27T16:41:52.6573169Z 
2022-09-27T16:41:52.6573893Z ----------------------------------------------------------------------
2022-09-27T16:41:52.6574609Z Ran 1 test in 7.085s
2022-09-27T16:41:52.6574894Z 
2022-09-27T16:41:52.6574996Z OK
2022-09-27T16:41:52.6575134Z 
2022-09-27T16:41:52.6575268Z Generating XML reports...
2022-09-27T16:41:52.6609792Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164145.xml
2022-09-27T16:41:52.8278020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:52.8279098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:52.8279891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:52.8280883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:52.8287071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:52.8288155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:53.4650133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:53.4650741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:53.4652330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:53.4653099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:54.0494112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:41:54.0516927Z 
2022-09-27T16:41:54.0517421Z Running tests...
2022-09-27T16:41:54.0517959Z ----------------------------------------------------------------------
2022-09-27T16:41:54.2073803Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmtunl_w
2022-09-27T16:41:54.2075144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmtunl_w/_remote_module_non_scriptable.py
2022-09-27T16:41:54.2798395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8eas1rq
2022-09-27T16:41:54.2799575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8eas1rq/_remote_module_non_scriptable.py
2022-09-27T16:41:54.5759387Z STAGE:2022-09-27 16:41:54 110448:110448 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:41:54.5772912Z STAGE:2022-09-27 16:41:54 110449:110449 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:41:54.7312607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:54.7313587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:54.7314774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:54.7315689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:55.1376134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:55.1377221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:41:55.1464787Z STAGE:2022-09-27 16:41:55 110449:110449 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:41:55 110448:110448 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:41:55.1465615Z 
2022-09-27T16:41:55.1703220Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:41:55.1707009Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:41:55.1748403Z STAGE:2022-09-27 16:41:55 110448:110448 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:41:55.1749649Z STAGE:2022-09-27 16:41:55 110449:110449 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:41:55.1791980Z STAGE:2022-09-27 16:41:55 110448:110448 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:41:55.1793179Z STAGE:2022-09-27 16:41:55 110449:110449 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:41:55.2372199Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:41:55.2396573Z 
2022-09-27T16:41:55.2397023Z Running tests...
2022-09-27T16:41:55.2397528Z ----------------------------------------------------------------------
2022-09-27T16:41:55.6953500Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:55.7155005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110615
2022-09-27T16:41:55.7161060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110616
2022-09-27T16:41:55.7999478Z ok (6.888s)
2022-09-27T16:41:55.7999850Z 
2022-09-27T16:41:55.8000903Z ----------------------------------------------------------------------
2022-09-27T16:41:55.8001567Z Ran 1 test in 6.889s
2022-09-27T16:41:55.8001895Z 
2022-09-27T16:41:55.8002079Z OK
2022-09-27T16:41:55.8002340Z 
2022-09-27T16:41:55.8002576Z Generating XML reports...
2022-09-27T16:41:55.8036852Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164148.xml
2022-09-27T16:41:56.9332595Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:41:56.9531543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110701
2022-09-27T16:41:56.9621235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110713
2022-09-27T16:41:57.3491005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:57.3491570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:57.3494365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:57.3494872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:57.4223399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:57.4223946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:57.4225402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:57.4225905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:57.9109154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:58.0102697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:58.0103492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:58.0105624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:58.0106107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:58.0351921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:58.0352650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:58.0425445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:58.0432871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:58.1366829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:58.5140864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:41:58.5164070Z 
2022-09-27T16:41:58.5164348Z Running tests...
2022-09-27T16:41:58.5164787Z ----------------------------------------------------------------------
2022-09-27T16:41:58.6659568Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:58.6660361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:58.6661206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:58.6661696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:58.6787916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:41:58.6788428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:41:58.6790777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:41:58.6791265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:41:59.1986778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:41:59.2226838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:41:59.2227676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:59.2292613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:41:59.2299497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:41:59.3243369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:41:59.7248378Z STAGE:2022-09-27 16:41:59 110615:110615 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:41:59.7997362Z STAGE:2022-09-27 16:41:59 110616:110616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:00.1818647Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:00.2028987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110803
2022-09-27T16:42:00.2034528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110804
2022-09-27T16:42:00.6018741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl1z0sgp
2022-09-27T16:42:00.6019366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl1z0sgp/_remote_module_non_scriptable.py
2022-09-27T16:42:00.6890072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlyg_7vs
2022-09-27T16:42:00.6891399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlyg_7vs/_remote_module_non_scriptable.py
2022-09-27T16:42:00.7198589Z [1664296920.719341] [28fd6574525b:110701:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:00.7199116Z [1664296920.719341] [28fd6574525b:110701:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:00.7261904Z [1664296920.725842] [28fd6574525b:110713:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:00.7262404Z [1664296920.725842] [28fd6574525b:110713:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:01.3801550Z STAGE:2022-09-27 16:42:01 110615:110615 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:01.3859413Z STAGE:2022-09-27 16:42:01 110616:110616 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:01.5533811Z STAGE:2022-09-27 16:42:01 110615:110615 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:01.5623629Z STAGE:2022-09-27 16:42:01 110616:110616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:01.5728374Z STAGE:2022-09-27 16:42:01 110616:110616 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:01.5784482Z STAGE:2022-09-27 16:42:01 110615:110615 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:01.6386039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:01.6386591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:01.6402624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:01.6405192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:01.8719510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:01.8720034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:01.8721586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:01.8722068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:01.9177997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:01.9178536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:01.9180238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:01.9180758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:02.3738451Z ok (7.134s)
2022-09-27T16:42:02.3738694Z 
2022-09-27T16:42:02.3739084Z ----------------------------------------------------------------------
2022-09-27T16:42:02.3739424Z Ran 1 test in 7.134s
2022-09-27T16:42:02.3739577Z 
2022-09-27T16:42:02.3739671Z OK
2022-09-27T16:42:02.3739806Z 
2022-09-27T16:42:02.3739938Z Generating XML reports...
2022-09-27T16:42:02.3850603Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164155.xml
2022-09-27T16:42:02.4773344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:02.4773931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:02.4774741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:02.4775449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:02.4883584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:02.5788940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:03.2332149Z ok (9.181s)
2022-09-27T16:42:03.2332458Z 
2022-09-27T16:42:03.2333014Z ----------------------------------------------------------------------
2022-09-27T16:42:03.2333360Z Ran 1 test in 9.181s
2022-09-27T16:42:03.2333526Z 
2022-09-27T16:42:03.2333622Z OK
2022-09-27T16:42:03.2333757Z 
2022-09-27T16:42:03.2333873Z Generating XML reports...
2022-09-27T16:42:03.2367855Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164154.xml
2022-09-27T16:42:03.9122127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r4rilsl
2022-09-27T16:42:03.9123406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r4rilsl/_remote_module_non_scriptable.py
2022-09-27T16:42:04.0616232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppd5eiohx
2022-09-27T16:42:04.0617173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppd5eiohx/_remote_module_non_scriptable.py
2022-09-27T16:42:04.3566247Z STAGE:2022-09-27 16:42:04 110804:110804 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:04.3643156Z STAGE:2022-09-27 16:42:04 110803:110803 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:04.5122555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:04.5123080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:04.5126951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:04.5127445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:04.9262502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:04.9263545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:04.9352525Z STAGE:2022-09-27 16:42:04 110803:110803 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:04.9353746Z STAGE:2022-09-27 16:42:04 110804:110804 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:05.0281927Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:42:05.0285253Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:42:05.0287528Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:42:05.0312478Z 
2022-09-27T16:42:05.0312993Z Running tests...
2022-09-27T16:42:05.0313877Z ----------------------------------------------------------------------
2022-09-27T16:42:05.0328810Z STAGE:2022-09-27 16:42:05 110803:110803 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:05.0330234Z STAGE:2022-09-27 16:42:05 110804:110804 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:05.0731861Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:42:05 110804:110804 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:42:05 110803:110803 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:05.0732935Z 
2022-09-27T16:42:05.3726938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:05.3727907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:05.3730008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:05.3730944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:05.7158566Z ok (7.199s)
2022-09-27T16:42:05.7159000Z 
2022-09-27T16:42:05.7159711Z ----------------------------------------------------------------------
2022-09-27T16:42:05.7160734Z Ran 1 test in 7.199s
2022-09-27T16:42:05.7161057Z 
2022-09-27T16:42:05.7161220Z OK
2022-09-27T16:42:05.7161501Z 
2022-09-27T16:42:05.7161769Z Generating XML reports...
2022-09-27T16:42:05.7195286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164158.xml
2022-09-27T16:42:05.8671560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:05.8694912Z 
2022-09-27T16:42:05.8695412Z Running tests...
2022-09-27T16:42:05.8695924Z ----------------------------------------------------------------------
2022-09-27T16:42:05.8714795Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:42:05.8715476Z 
2022-09-27T16:42:05.8715843Z ----------------------------------------------------------------------
2022-09-27T16:42:05.8716374Z Ran 1 test in 0.002s
2022-09-27T16:42:05.8716547Z 
2022-09-27T16:42:05.8716658Z OK (skipped=1)
2022-09-27T16:42:05.8716822Z 
2022-09-27T16:42:05.8716953Z Generating XML reports...
2022-09-27T16:42:05.8752388Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164205.xml
2022-09-27T16:42:06.7091642Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:06.7292115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110979
2022-09-27T16:42:06.7298627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110980
2022-09-27T16:42:07.8289315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:07.8289888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:07.8291319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:07.8291823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:07.8547585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:07.8548080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:07.8551571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:07.8552061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:08.3070049Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:08.3092811Z 
2022-09-27T16:42:08.3092973Z Running tests...
2022-09-27T16:42:08.3093432Z ----------------------------------------------------------------------
2022-09-27T16:42:08.3111879Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:42:08.3112826Z 
2022-09-27T16:42:08.3113118Z ----------------------------------------------------------------------
2022-09-27T16:42:08.3113472Z Ran 1 test in 0.002s
2022-09-27T16:42:08.3113641Z 
2022-09-27T16:42:08.3113756Z OK (skipped=1)
2022-09-27T16:42:08.3113915Z 
2022-09-27T16:42:08.3114254Z Generating XML reports...
2022-09-27T16:42:08.3148537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164208.xml
2022-09-27T16:42:08.3602857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:42:08.3627302Z 
2022-09-27T16:42:08.3627445Z Running tests...
2022-09-27T16:42:08.3628210Z ----------------------------------------------------------------------
2022-09-27T16:42:08.4131825Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:08.4132438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:08.4133576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:08.4134055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:08.4134653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:08.4135119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:08.4138162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:08.4138636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:08.9612909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:08.9623688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:08.9624741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:08.9714489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:08.9721756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:09.0639757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:09.5371126Z skip: Need at least 4 CUDA devices (4.506s)
2022-09-27T16:42:09.5371386Z 
2022-09-27T16:42:09.5371997Z ----------------------------------------------------------------------
2022-09-27T16:42:09.5372368Z Ran 1 test in 4.506s
2022-09-27T16:42:09.5372538Z 
2022-09-27T16:42:09.5372652Z OK (skipped=1)
2022-09-27T16:42:09.5372824Z 
2022-09-27T16:42:09.5372936Z Generating XML reports...
2022-09-27T16:42:09.5573844Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164205.xml
2022-09-27T16:42:10.0013338Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:10.0213789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111146
2022-09-27T16:42:10.0219881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111147
2022-09-27T16:42:10.2749797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:10.2750305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:10.2752567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:10.2753074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:10.8112931Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:10.8135128Z 
2022-09-27T16:42:10.8135504Z Running tests...
2022-09-27T16:42:10.8135954Z ----------------------------------------------------------------------
2022-09-27T16:42:11.6266549Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:11.6267187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:11.6268640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:11.6269312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:11.6439438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:11.6439930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:11.6442944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:11.6443630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:11.6921705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:11.6922228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:11.6923292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:11.6923935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:12.1501497Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:42:12.1524182Z 
2022-09-27T16:42:12.1524469Z Running tests...
2022-09-27T16:42:12.1525021Z ----------------------------------------------------------------------
2022-09-27T16:42:12.2342736Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:12.2343512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:12.2344513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:12.2345337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:12.2352522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:12.2353305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:12.3698591Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:12.3904303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111258
2022-09-27T16:42:12.3908997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111259
2022-09-27T16:42:13.6590935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbmkdv6i
2022-09-27T16:42:13.6591579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbmkdv6i/_remote_module_non_scriptable.py
2022-09-27T16:42:13.6802193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jrnb7yw
2022-09-27T16:42:13.6804012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jrnb7yw/_remote_module_non_scriptable.py
2022-09-27T16:42:13.9196103Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:13.9404911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111323
2022-09-27T16:42:13.9409418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111324
2022-09-27T16:42:14.0564407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:14.0564934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:14.0566327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:14.0566824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:14.0700571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:14.0701075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:14.0703330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:14.0703828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:14.2323506Z ok (5.869s)
2022-09-27T16:42:14.2323739Z 
2022-09-27T16:42:14.2324108Z ----------------------------------------------------------------------
2022-09-27T16:42:14.2324431Z Ran 1 test in 5.870s
2022-09-27T16:42:14.2324606Z 
2022-09-27T16:42:14.2324702Z OK
2022-09-27T16:42:14.2324838Z 
2022-09-27T16:42:14.2324978Z Generating XML reports...
2022-09-27T16:42:14.2359967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164208.xml
2022-09-27T16:42:14.5882732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:14.5973432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:14.5976003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:14.5984073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:14.6288782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:14.6993025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:15.6586407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:15.6586924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:15.6588037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:15.6588513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:15.7377579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:15.7378108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:15.7379557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:15.7380036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:16.1975454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:16.2723998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:16.2724835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:16.2786710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:16.2793825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:16.3738109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:16.5303928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:16.5304435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:16.5306624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:16.5307099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:16.8482712Z skip: Need at least 4 CUDA devices (4.695s)
2022-09-27T16:42:16.8482969Z 
2022-09-27T16:42:16.8483334Z ----------------------------------------------------------------------
2022-09-27T16:42:16.8483681Z Ran 1 test in 4.696s
2022-09-27T16:42:16.8483848Z 
2022-09-27T16:42:16.8483958Z OK (skipped=1)
2022-09-27T16:42:16.8484133Z 
2022-09-27T16:42:16.8484263Z Generating XML reports...
2022-09-27T16:42:16.8520261Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164212.xml
2022-09-27T16:42:17.0761356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:42:17.0786784Z 
2022-09-27T16:42:17.0787406Z Running tests...
2022-09-27T16:42:17.0787975Z ----------------------------------------------------------------------
2022-09-27T16:42:17.9032288Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... ok (7.089s)
2022-09-27T16:42:17.9032873Z 
2022-09-27T16:42:17.9033535Z ----------------------------------------------------------------------
2022-09-27T16:42:17.9034172Z Ran 1 test in 7.089s
2022-09-27T16:42:17.9034492Z 
2022-09-27T16:42:17.9034672Z OK
2022-09-27T16:42:17.9034937Z 
2022-09-27T16:42:17.9035177Z Generating XML reports...
2022-09-27T16:42:17.9072666Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164210.xml
2022-09-27T16:42:18.8912313Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:18.9105465Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111480
2022-09-27T16:42:18.9112146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111481
2022-09-27T16:42:19.0609763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:19.0612487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:19.0618737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:19.0621819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:19.5680130Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:42:19.5702806Z 
2022-09-27T16:42:19.5703612Z Running tests...
2022-09-27T16:42:19.5704129Z ----------------------------------------------------------------------
2022-09-27T16:42:20.0708343Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:20.0708993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:20.0710199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:20.0710671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:20.5783646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:20.5809239Z 
2022-09-27T16:42:20.5809674Z Running tests...
2022-09-27T16:42:20.5810170Z ----------------------------------------------------------------------
2022-09-27T16:42:20.5980823Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:20.5981716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:20.5984168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:20.5984672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:20.6291871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:20.6292363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:20.6294169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:20.6294670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:21.1778067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:21.1779203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:21.1780751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:21.1782214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:21.1859325Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:21.1886518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:21.2059110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111590
2022-09-27T16:42:21.2064571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111591
2022-09-27T16:42:21.2793187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:22.3006618Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:22.3186101Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.737s)
2022-09-27T16:42:22.3187320Z 
2022-09-27T16:42:22.3187885Z ----------------------------------------------------------------------
2022-09-27T16:42:22.3188504Z Ran 1 test in 1.737s
2022-09-27T16:42:22.3188816Z 
2022-09-27T16:42:22.3189025Z OK (skipped=1)
2022-09-27T16:42:22.3189329Z 
2022-09-27T16:42:22.3189565Z Generating XML reports...
2022-09-27T16:42:22.3224033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164220.xml
2022-09-27T16:42:22.5785623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8lm_l19
2022-09-27T16:42:22.5786666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8lm_l19/_remote_module_non_scriptable.py
2022-09-27T16:42:22.6503468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_6z3yuw
2022-09-27T16:42:22.6504462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_6z3yuw/_remote_module_non_scriptable.py
2022-09-27T16:42:22.6694206Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:42:22.6695673Z   warnings.warn(
2022-09-27T16:42:22.6697735Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:42:22.6699428Z   warnings.warn(
2022-09-27T16:42:22.8782462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:22.8782975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:22.8784439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:22.8784915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:22.8830684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:22.8831185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:22.8834588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:22.8835073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:23.1580600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:23.1581152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:23.1819828Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:42:23.1821420Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:42:23.1993312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:23.1993831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:23.2093892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:23.2094710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:23.4126494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:23.4134506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:23.4135313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:23.4228608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:23.4235358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:23.5174209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:23.8231964Z ok (6.744s)
2022-09-27T16:42:23.8232220Z 
2022-09-27T16:42:23.8232631Z ----------------------------------------------------------------------
2022-09-27T16:42:23.8232972Z Ran 1 test in 6.744s
2022-09-27T16:42:23.8233142Z 
2022-09-27T16:42:23.8233242Z OK
2022-09-27T16:42:23.8233367Z 
2022-09-27T16:42:23.8233770Z Generating XML reports...
2022-09-27T16:42:23.8268094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164217.xml
2022-09-27T16:42:24.5045905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:24.5046458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:24.5048212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:24.5048707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:24.8414224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlwyr86b
2022-09-27T16:42:24.8415821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlwyr86b/_remote_module_non_scriptable.py
2022-09-27T16:42:24.9247826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxuioxze
2022-09-27T16:42:24.9249644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxuioxze/_remote_module_non_scriptable.py
2022-09-27T16:42:24.9517483Z [1664296944.951419] [28fd6574525b:111590:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:24.9517994Z [1664296944.951419] [28fd6574525b:111590:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:24.9571855Z [1664296944.956900] [28fd6574525b:111591:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:24.9572331Z [1664296944.956900] [28fd6574525b:111591:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:25.0234767Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:25.0257956Z 
2022-09-27T16:42:25.0258540Z Running tests...
2022-09-27T16:42:25.0259270Z ----------------------------------------------------------------------
2022-09-27T16:42:25.7840886Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:25.7841554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:25.9688794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:25.9689686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:25.9691609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:25.9692732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:26.4178349Z ok (6.847s)
2022-09-27T16:42:26.4178699Z 
2022-09-27T16:42:26.4179327Z ----------------------------------------------------------------------
2022-09-27T16:42:26.4179966Z Ran 1 test in 6.847s
2022-09-27T16:42:26.4180267Z 
2022-09-27T16:42:26.4180427Z OK
2022-09-27T16:42:26.4180662Z 
2022-09-27T16:42:26.4180885Z Generating XML reports...
2022-09-27T16:42:26.4216579Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164219.xml
2022-09-27T16:42:26.4886013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:42:26.4909863Z 
2022-09-27T16:42:26.4910439Z Running tests...
2022-09-27T16:42:26.4910962Z ----------------------------------------------------------------------
2022-09-27T16:42:26.6563169Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:26.6767941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111748
2022-09-27T16:42:26.6773087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111749
2022-09-27T16:42:28.2279436Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:28.2486628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111845
2022-09-27T16:42:28.2492811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111846
2022-09-27T16:42:28.3135719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:28.3136713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:28.3138402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:28.3139319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:28.3699404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:28.3700393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:28.3701597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:28.3702555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:28.5987186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:28.5988193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:28.5989366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:28.5990299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:28.8207145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:28.8778125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:28.8780581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:28.8820087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:28.8837861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:28.8839477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:42:28.9788571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:28.9791858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:42:28.9792849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:28.9856299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:29.1522914Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:42:29.1547343Z 
2022-09-27T16:42:29.1547725Z Running tests...
2022-09-27T16:42:29.1548209Z ----------------------------------------------------------------------
2022-09-27T16:42:29.4847730Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... ok (4.458s)
2022-09-27T16:42:29.4848334Z 
2022-09-27T16:42:29.4848989Z ----------------------------------------------------------------------
2022-09-27T16:42:29.4849686Z Ran 1 test in 4.459s
2022-09-27T16:42:29.4850010Z 
2022-09-27T16:42:29.4850196Z OK
2022-09-27T16:42:29.4850454Z 
2022-09-27T16:42:29.4850693Z Generating XML reports...
2022-09-27T16:42:29.4882102Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164225.xml
2022-09-27T16:42:29.8888596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:29.8889503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:29.8890982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:29.8891621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:29.9121907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:29.9122397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:29.9125186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:29.9125680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:30.4737944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:30.4738548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:30.4739341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:30.4740054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:30.4746775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:30.4747279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:30.8646369Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:30.8851272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111968
2022-09-27T16:42:30.8856785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111969
2022-09-27T16:42:31.6033009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:31.6034115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:31.6037552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:31.6038622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:31.9340112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw067c9f
2022-09-27T16:42:31.9340751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw067c9f/_remote_module_non_scriptable.py
2022-09-27T16:42:31.9359926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3px380u
2022-09-27T16:42:31.9362753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3px380u/_remote_module_non_scriptable.py
2022-09-27T16:42:31.9553252Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:42:31.9554044Z   warnings.warn(
2022-09-27T16:42:31.9555119Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-09-27T16:42:31.9555852Z   warnings.warn(
2022-09-27T16:42:32.1382700Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:32.1405130Z 
2022-09-27T16:42:32.1405332Z Running tests...
2022-09-27T16:42:32.1406100Z ----------------------------------------------------------------------
2022-09-27T16:42:32.3969036Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:32.3969665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:32.5544776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:32.5545299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:32.5546160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:32.5546669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:32.5838507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:32.5839056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:32.5841366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:32.5841861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:32.9596618Z ok (6.468s)
2022-09-27T16:42:32.9596900Z 
2022-09-27T16:42:32.9597304Z ----------------------------------------------------------------------
2022-09-27T16:42:32.9597650Z Ran 1 test in 6.469s
2022-09-27T16:42:32.9597820Z 
2022-09-27T16:42:32.9597918Z OK
2022-09-27T16:42:32.9598036Z 
2022-09-27T16:42:32.9598171Z Generating XML reports...
2022-09-27T16:42:32.9633369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164226.xml
2022-09-27T16:42:33.0820532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:33.1230840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:33.1231664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:33.1329375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:33.1335707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:33.1338982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:42:33.2244790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:33.2248452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:42:33.2249284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:33.2253876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:33.2345245Z STAGE:2022-09-27 16:42:33 111968:111968 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:42:33 111969:111969 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:33.2345661Z 
2022-09-27T16:42:33.8515296Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:33.8717852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112052
2022-09-27T16:42:33.8722981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112053
2022-09-27T16:42:34.9601391Z [1664296954.959551] [28fd6574525b:111968:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:34.9601968Z [1664296954.959551] [28fd6574525b:111968:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:35.0272676Z [1664296955.026779] [28fd6574525b:111969:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:35.0273177Z [1664296955.026779] [28fd6574525b:111969:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:35.1457590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:35.1458380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:35.1459805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:35.1460290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:35.4867565Z STAGE:2022-09-27 16:42:35 111969:111969 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:42:35 111968:111968 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:35.4868075Z 
2022-09-27T16:42:35.5595874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:35.5596489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:35.5598053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:35.5598550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:35.6107944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:35.6108442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:35.6112021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:35.6112513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:35.6117696Z STAGE:2022-09-27 16:42:35 111968:111968 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:35.6129205Z STAGE:2022-09-27 16:42:35 111968:111968 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:35.6133851Z STAGE:2022-09-27 16:42:35 111969:111969 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:35.6144518Z STAGE:2022-09-27 16:42:35 111969:111969 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:35.6453113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:42:35.6476791Z 
2022-09-27T16:42:35.6476945Z Running tests...
2022-09-27T16:42:35.6478134Z ----------------------------------------------------------------------
2022-09-27T16:42:36.0718799Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:36.1226907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:36.1228514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:36.1229955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:36.1234630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:36.1235650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:36.6995141Z ok (7.544s)
2022-09-27T16:42:36.6995522Z 
2022-09-27T16:42:36.6996238Z ----------------------------------------------------------------------
2022-09-27T16:42:36.6996911Z Ran 1 test in 7.545s
2022-09-27T16:42:36.6997232Z 
2022-09-27T16:42:36.6997394Z OK
2022-09-27T16:42:36.6997669Z 
2022-09-27T16:42:36.6998319Z Generating XML reports...
2022-09-27T16:42:36.7032789Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164229.xml
2022-09-27T16:42:37.3865056Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:37.4069569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112166
2022-09-27T16:42:37.4074608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112167
2022-09-27T16:42:38.9787737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:38.9788554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:38.9790136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:38.9790645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:39.1331814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:39.1332349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:39.1333259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:39.1333757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:39.1575799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:39.1576270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:39.1579348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:39.1579851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:39.2842097Z ok (7.143s)
2022-09-27T16:42:39.2842301Z 
2022-09-27T16:42:39.2842702Z ----------------------------------------------------------------------
2022-09-27T16:42:39.2843047Z Ran 1 test in 7.144s
2022-09-27T16:42:39.2843197Z 
2022-09-27T16:42:39.2843301Z OK
2022-09-27T16:42:39.2843438Z 
2022-09-27T16:42:39.2843574Z Generating XML reports...
2022-09-27T16:42:39.2879561Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164232.xml
2022-09-27T16:42:39.5006196Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:42:39.5030497Z 
2022-09-27T16:42:39.5031232Z Running tests...
2022-09-27T16:42:39.5032040Z ----------------------------------------------------------------------
2022-09-27T16:42:39.6822526Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:39.6823215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:39.6824023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:39.6824712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:39.6831190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:39.6831718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:41.0306692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtw_fdfp
2022-09-27T16:42:41.0309049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtw_fdfp/_remote_module_non_scriptable.py
2022-09-27T16:42:41.0420579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbk3jkqr
2022-09-27T16:42:41.0423104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbk3jkqr/_remote_module_non_scriptable.py
2022-09-27T16:42:41.2421300Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:41.2628183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112316
2022-09-27T16:42:41.2634190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112317
2022-09-27T16:42:41.3872068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:41.3872830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:41.3873453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:41.3873960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:41.5198382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:41.5198931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:41.5300549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:41.5301053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:41.9225067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:41.9264722Z 
2022-09-27T16:42:41.9265106Z Running tests...
2022-09-27T16:42:41.9266020Z ----------------------------------------------------------------------
2022-09-27T16:42:42.2184047Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... ok (6.570s)
2022-09-27T16:42:42.2184344Z 
2022-09-27T16:42:42.2184728Z ----------------------------------------------------------------------
2022-09-27T16:42:42.2185094Z Ran 1 test in 6.570s
2022-09-27T16:42:42.2185263Z 
2022-09-27T16:42:42.2185358Z OK
2022-09-27T16:42:42.2185494Z 
2022-09-27T16:42:42.2185621Z Generating XML reports...
2022-09-27T16:42:42.2221170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164235.xml
2022-09-27T16:42:42.8686254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:42.8686779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:42.8690188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:42.8690674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:42.9098615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:42.9099422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:42.9101911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:42.9102400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:43.3869795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:43.4896663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:43.4898290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:43.4988667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:43.4997061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:43.5001141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:42:43.5908616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:43.5911370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:42:43.5912881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:43.5919337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:43.5952742Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:43.5978476Z STAGE:2022-09-27 16:42:43 112316:112316 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:43.6012987Z STAGE:2022-09-27 16:42:43 112317:112317 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:43.6195379Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112428
2022-09-27T16:42:43.6201377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112429
2022-09-27T16:42:44.3369803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:44.3370795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:44.3374888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:44.3376608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:44.8602222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:42:44.8625068Z 
2022-09-27T16:42:44.8625565Z Running tests...
2022-09-27T16:42:44.8626421Z ----------------------------------------------------------------------
2022-09-27T16:42:45.2828513Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:45.2829674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:45.2830877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:45.2831807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:45.3197886Z [1664296965.319216] [28fd6574525b:112317:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:45.3198911Z [1664296965.319216] [28fd6574525b:112317:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:45.3268534Z [1664296965.326448] [28fd6574525b:112316:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:45.3269839Z [1664296965.326448] [28fd6574525b:112316:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:45.3383971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:45.3384966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:45.3387415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:45.3388370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:45.7168971Z STAGE:2022-09-27 16:42:45 112317:112317 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:42:45 112316:112316 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:45.7169892Z 
2022-09-27T16:42:45.8131980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:45.8410286Z STAGE:2022-09-27 16:42:45 112317:112317 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:45.8438497Z STAGE:2022-09-27 16:42:45 112316:112316 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:45.8448979Z STAGE:2022-09-27 16:42:45 112317:112317 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:45.8449601Z STAGE:2022-09-27 16:42:45 112316:112316 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:45.8657134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:45.8657945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:45.8743260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:45.8750671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:45.9671467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:46.4779635Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:46.4979772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112511
2022-09-27T16:42:46.4985590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112512
2022-09-27T16:42:46.5271687Z ok (4.600s)
2022-09-27T16:42:46.5271869Z 
2022-09-27T16:42:46.5272834Z ----------------------------------------------------------------------
2022-09-27T16:42:46.5273300Z Ran 1 test in 4.601s
2022-09-27T16:42:46.5273477Z 
2022-09-27T16:42:46.5273575Z OK
2022-09-27T16:42:46.5273728Z 
2022-09-27T16:42:46.5273861Z Generating XML reports...
2022-09-27T16:42:46.5310553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164241.xml
2022-09-27T16:42:47.0759083Z ok (7.573s)
2022-09-27T16:42:47.0759299Z 
2022-09-27T16:42:47.0760148Z ----------------------------------------------------------------------
2022-09-27T16:42:47.0760763Z Ran 1 test in 7.573s
2022-09-27T16:42:47.0760945Z 
2022-09-27T16:42:47.0761035Z OK
2022-09-27T16:42:47.0761185Z 
2022-09-27T16:42:47.0761347Z Generating XML reports...
2022-09-27T16:42:47.0798100Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164239.xml
2022-09-27T16:42:48.1889286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:48.1889845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:48.1892693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:48.1893199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:48.2434931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:48.2435498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:48.2438417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:48.2438930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:48.7691833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:48.7692947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:48.7694402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:48.7695762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:48.7799742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:48.7835758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:48.7836508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:48.7840300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:48.7841447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:48.8706131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:49.2494336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:49.2494848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:49.2497426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:49.2497959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:49.3268323Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:49.3293766Z 
2022-09-27T16:42:49.3294032Z Running tests...
2022-09-27T16:42:49.3294482Z ----------------------------------------------------------------------
2022-09-27T16:42:49.7625223Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:42:49.7649708Z 
2022-09-27T16:42:49.7650187Z Running tests...
2022-09-27T16:42:49.7650667Z ----------------------------------------------------------------------
2022-09-27T16:42:50.1731298Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdo1mn0hm
2022-09-27T16:42:50.1732033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdo1mn0hm/_remote_module_non_scriptable.py
2022-09-27T16:42:50.2027689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqeouml6
2022-09-27T16:42:50.2028502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqeouml6/_remote_module_non_scriptable.py
2022-09-27T16:42:50.4998108Z STAGE:2022-09-27 16:42:50 112511:112511 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:50.5004629Z STAGE:2022-09-27 16:42:50 112512:112512 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:50.5095691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:50.5096389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:42:50.5304985Z STAGE:2022-09-27 16:42:50 112512:112512 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:42:50 112511:112511 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:50.5305435Z 
2022-09-27T16:42:50.5944864Z STAGE:2022-09-27 16:42:50 112511:112511 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:50.9771990Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:50.9983294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112664
2022-09-27T16:42:50.9988678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112665
2022-09-27T16:42:51.3694735Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:51.3892269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112667
2022-09-27T16:42:51.3898188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112668
2022-09-27T16:42:51.4177542Z STAGE:2022-09-27 16:42:51 112511:112511 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:52.1117379Z ok (7.249s)
2022-09-27T16:42:52.1126155Z 
2022-09-27T16:42:52.1127587Z ----------------------------------------------------------------------
2022-09-27T16:42:52.1128742Z Ran 1 test in 7.250s
2022-09-27T16:42:52.1129388Z 
2022-09-27T16:42:52.1129758Z OK
2022-09-27T16:42:52.1130611Z 
2022-09-27T16:42:52.1130893Z Generating XML reports...
2022-09-27T16:42:52.1195574Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164244.xml
2022-09-27T16:42:52.7151141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:52.7151657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:52.7153884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:52.7154466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:52.7717642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:52.7718159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:52.7719454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:52.7719963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:53.1157055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:53.1157561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:53.1159080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:53.1159652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:53.1598107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:53.1598620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:53.1601357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:53.1601864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:53.2201453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:53.2747517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:53.2748335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:53.2810815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:53.2818375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:53.2822268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:42:53.3763107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:53.3771826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:42:53.3775004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:53.3840134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:53.6860826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:53.7088839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:53.7090288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:53.7166040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:53.7173248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:53.7176416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:42:53.8055052Z ok (4.476s)
2022-09-27T16:42:53.8055317Z 
2022-09-27T16:42:53.8055724Z ----------------------------------------------------------------------
2022-09-27T16:42:53.8056047Z Ran 1 test in 4.476s
2022-09-27T16:42:53.8056214Z 
2022-09-27T16:42:53.8056310Z OK
2022-09-27T16:42:53.8056456Z 
2022-09-27T16:42:53.8056593Z Generating XML reports...
2022-09-27T16:42:53.8091735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164249.xml
2022-09-27T16:42:53.8099982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:53.8104080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:42:53.8104874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:53.8170361Z STAGE:2022-09-27 16:42:53 112667:112667 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:53.8190915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:42:53.8250363Z STAGE:2022-09-27 16:42:53 112668:112668 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:54.3291852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:54.3292635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:54.3293866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:54.3294397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:54.8400535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:42:54.8424146Z 
2022-09-27T16:42:54.8424483Z Running tests...
2022-09-27T16:42:54.8424932Z ----------------------------------------------------------------------
2022-09-27T16:42:55.4762508Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... [1664296975.475721] [28fd6574525b:112667:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:55.4763142Z [1664296975.475721] [28fd6574525b:112667:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:55.5100478Z [1664296975.509565] [28fd6574525b:112668:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:42:55.5100995Z [1664296975.509565] [28fd6574525b:112668:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:42:55.8970055Z STAGE:2022-09-27 16:42:55 112667:112667 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:42:55 112668:112668 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:55.8970537Z 
2022-09-27T16:42:55.9396443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:55.9397007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:55.9399476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:55.9399976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:56.0292496Z STAGE:2022-09-27 16:42:56 112668:112668 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:56.0531870Z STAGE:2022-09-27 16:42:56 112667:112667 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:42:56.0542246Z STAGE:2022-09-27 16:42:56 112668:112668 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:42:56 112667:112667 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:42:56.0542660Z 
2022-09-27T16:42:56.4464484Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:56.4659171Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112890
2022-09-27T16:42:56.4665624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112891
2022-09-27T16:42:56.4785662Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:42:56.4810343Z 
2022-09-27T16:42:56.4810774Z Running tests...
2022-09-27T16:42:56.4811288Z ----------------------------------------------------------------------
2022-09-27T16:42:57.4043203Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... ok (7.639s)
2022-09-27T16:42:57.4043555Z 
2022-09-27T16:42:57.4043972Z ----------------------------------------------------------------------
2022-09-27T16:42:57.4044311Z Ran 1 test in 7.639s
2022-09-27T16:42:57.4044482Z 
2022-09-27T16:42:57.4044576Z OK
2022-09-27T16:42:57.4044694Z 
2022-09-27T16:42:57.4044823Z Generating XML reports...
2022-09-27T16:42:57.4079530Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164249.xml
2022-09-27T16:42:58.0940484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:58.0941008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:58.0942175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:58.0942686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:58.0951232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:58.0951711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:58.0955251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:58.0955740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:58.1991972Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:42:58.2197470Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112959
2022-09-27T16:42:58.2203955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112960
2022-09-27T16:42:58.6260634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:42:58.6262238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:42:58.6264022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:58.6265657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:42:58.6451008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:42:58.7278848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:42:59.6412350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:59.6413262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:59.6415318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:59.6416301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:59.9152179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:59.9153552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:59.9154686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:59.9155600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:42:59.9701399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:42:59.9702322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:42:59.9703529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:42:59.9704491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:00.0555522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglx0nr6y
2022-09-27T16:43:00.0556719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglx0nr6y/_remote_module_non_scriptable.py
2022-09-27T16:43:00.0931446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6op1zvy
2022-09-27T16:43:00.0932339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6op1zvy/_remote_module_non_scriptable.py
2022-09-27T16:43:00.1675474Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:43:00.1701229Z 
2022-09-27T16:43:00.1701565Z Running tests...
2022-09-27T16:43:00.1702409Z ----------------------------------------------------------------------
2022-09-27T16:43:00.4541530Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:00.5101788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:00.5103258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:00.5149572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:00.5156276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:00.5159844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:43:00.6115267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:00.6118010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:43:00.6119056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:00.6173276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:00.6768448Z ok (5.834s)
2022-09-27T16:43:00.6768656Z 
2022-09-27T16:43:00.6769029Z ----------------------------------------------------------------------
2022-09-27T16:43:00.6769376Z Ran 1 test in 5.834s
2022-09-27T16:43:00.6769548Z 
2022-09-27T16:43:00.6769646Z OK
2022-09-27T16:43:00.6769766Z 
2022-09-27T16:43:00.6769904Z Generating XML reports...
2022-09-27T16:43:00.6806035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164254.xml
2022-09-27T16:43:01.1277951Z ok (4.646s)
2022-09-27T16:43:01.1278164Z 
2022-09-27T16:43:01.1278558Z ----------------------------------------------------------------------
2022-09-27T16:43:01.1278884Z Ran 1 test in 4.647s
2022-09-27T16:43:01.1279056Z 
2022-09-27T16:43:01.1279167Z OK
2022-09-27T16:43:01.1279305Z 
2022-09-27T16:43:01.1279444Z Generating XML reports...
2022-09-27T16:43:01.1313948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164256.xml
2022-09-27T16:43:01.8937520Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:01.9532103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113112
2022-09-27T16:43:01.9768866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113113
2022-09-27T16:43:02.9770043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:02.9770548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:02.9773403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:02.9773921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:03.3188624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:03.3189160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:03.3191183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:03.3191681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:03.4742991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:43:03.4767343Z 
2022-09-27T16:43:03.4767617Z Running tests...
2022-09-27T16:43:03.4768240Z ----------------------------------------------------------------------
2022-09-27T16:43:03.6122918Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:03.6123535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:03.6125530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:03.6126057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:03.6573946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:03.6574442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:03.6576854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:03.6577352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:03.8086812Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:43:03.8109347Z 
2022-09-27T16:43:03.8110026Z Running tests...
2022-09-27T16:43:03.8110558Z ----------------------------------------------------------------------
2022-09-27T16:43:04.1288039Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:04.1724778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:04.1725571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:04.1794427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:04.1803374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:04.1806752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:43:04.2738384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:04.2740609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:43:04.2741725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:04.2806879Z STAGE:2022-09-27 16:43:04 113113:113113 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:04.2821846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:04.2880684Z STAGE:2022-09-27 16:43:04 113112:113112 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:05.1718298Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:05.1927604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113224
2022-09-27T16:43:05.1933735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113225
2022-09-27T16:43:05.4607475Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:05.4809813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113227
2022-09-27T16:43:05.4814643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113228
2022-09-27T16:43:05.9750209Z [1664296985.974366] [28fd6574525b:113113:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:43:05.9751407Z [1664296985.974366] [28fd6574525b:113113:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:43:06.0152247Z [1664296986.014728] [28fd6574525b:113112:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:43:06.0152761Z [1664296986.014728] [28fd6574525b:113112:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:43:06.4724275Z STAGE:2022-09-27 16:43:06 113113:113113 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:43:06 113112:113112 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:06.4725154Z 
2022-09-27T16:43:06.5964413Z STAGE:2022-09-27 16:43:06 113113:113113 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:06.5995982Z STAGE:2022-09-27 16:43:06 113112:113112 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:06.6008434Z STAGE:2022-09-27 16:43:06 113113:113113 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:06.6009675Z STAGE:2022-09-27 16:43:06 113112:113112 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:06.9042576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:06.9043616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:06.9045263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:06.9046255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:06.9400710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:06.9401679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:06.9402926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:06.9403465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:07.1680747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:07.1681278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:07.1683396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:07.1683905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:07.2017265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:07.2018053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:07.2021404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:07.2021910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:07.4598984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:07.4599529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:07.4600827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:07.4601627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:07.4709377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:07.5613195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:07.6756469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:07.7351318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:07.7352144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:07.7365124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:07.7372043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:07.7890246Z ok (7.619s)
2022-09-27T16:43:07.7890464Z 
2022-09-27T16:43:07.7890857Z ----------------------------------------------------------------------
2022-09-27T16:43:07.7891221Z Ran 1 test in 7.619s
2022-09-27T16:43:07.7891389Z 
2022-09-27T16:43:07.7891465Z OK
2022-09-27T16:43:07.7891610Z 
2022-09-27T16:43:07.7891743Z Generating XML reports...
2022-09-27T16:43:07.7928816Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164300.xml
2022-09-27T16:43:07.8367990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:08.9527481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3ln4hbs
2022-09-27T16:43:08.9528695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3ln4hbs/_remote_module_non_scriptable.py
2022-09-27T16:43:09.0369293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzebowevc
2022-09-27T16:43:09.0370726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzebowevc/_remote_module_non_scriptable.py
2022-09-27T16:43:09.1192145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwc22nse
2022-09-27T16:43:09.1193440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwc22nse/_remote_module_non_scriptable.py
2022-09-27T16:43:09.2683411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthgtup0_
2022-09-27T16:43:09.2685359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthgtup0_/_remote_module_non_scriptable.py
2022-09-27T16:43:09.6038429Z ok (6.127s)
2022-09-27T16:43:09.6038640Z 
2022-09-27T16:43:09.6039046Z ----------------------------------------------------------------------
2022-09-27T16:43:09.6039393Z Ran 1 test in 6.127s
2022-09-27T16:43:09.6039561Z 
2022-09-27T16:43:09.6039660Z OK
2022-09-27T16:43:09.6039800Z 
2022-09-27T16:43:09.6039941Z Generating XML reports...
2022-09-27T16:43:09.6085017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164303.xml
2022-09-27T16:43:10.0842609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:10.0843230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:10.0844544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:10.0845347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:10.3925473Z ok (6.581s)
2022-09-27T16:43:10.3925698Z 
2022-09-27T16:43:10.3926099Z ----------------------------------------------------------------------
2022-09-27T16:43:10.3926450Z Ran 1 test in 6.581s
2022-09-27T16:43:10.3926604Z 
2022-09-27T16:43:10.3926705Z OK
2022-09-27T16:43:10.3926841Z 
2022-09-27T16:43:10.3927006Z Generating XML reports...
2022-09-27T16:43:10.3960769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164303.xml
2022-09-27T16:43:10.6137109Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:43:10.6160416Z 
2022-09-27T16:43:10.6160774Z Running tests...
2022-09-27T16:43:10.6161266Z ----------------------------------------------------------------------
2022-09-27T16:43:11.8569857Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:11.8570457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:11.8571263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:11.8571773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:12.3196578Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:12.3394010Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113491
2022-09-27T16:43:12.3400879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113492
2022-09-27T16:43:12.3614565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:43:12.3638928Z 
2022-09-27T16:43:12.3639094Z Running tests...
2022-09-27T16:43:12.3639758Z ----------------------------------------------------------------------
2022-09-27T16:43:12.5708700Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:12.5709461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:12.5711381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:12.5711877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:13.1483117Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:43:13.1520874Z 
2022-09-27T16:43:13.1521146Z Running tests...
2022-09-27T16:43:13.1521594Z ----------------------------------------------------------------------
2022-09-27T16:43:13.1562694Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.004s)
2022-09-27T16:43:13.1563383Z 
2022-09-27T16:43:13.1563665Z ----------------------------------------------------------------------
2022-09-27T16:43:13.1564024Z Ran 1 test in 0.004s
2022-09-27T16:43:13.1564192Z 
2022-09-27T16:43:13.1564303Z OK (skipped=1)
2022-09-27T16:43:13.1564461Z 
2022-09-27T16:43:13.1564588Z Generating XML reports...
2022-09-27T16:43:13.1618407Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164313.xml
2022-09-27T16:43:14.0076177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:14.0077337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:14.0078197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:14.0078666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:14.0142463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:14.0142941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:14.0146468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:14.0146939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:14.0233869Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:14.0436823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113561
2022-09-27T16:43:14.0443963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113562
2022-09-27T16:43:14.5694094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:14.5765672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:14.5766474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:14.5794834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:14.5801411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:14.6781769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:15.1470176Z skip: Skipped due to small world size. (4.531s)
2022-09-27T16:43:15.1470434Z 
2022-09-27T16:43:15.1470836Z ----------------------------------------------------------------------
2022-09-27T16:43:15.1471183Z Ran 1 test in 4.531s
2022-09-27T16:43:15.1471348Z 
2022-09-27T16:43:15.1471458Z OK (skipped=1)
2022-09-27T16:43:15.1471614Z 
2022-09-27T16:43:15.1471724Z Generating XML reports...
2022-09-27T16:43:15.1507161Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164310.xml
2022-09-27T16:43:15.1914153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:15.1914647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:15.1916663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:15.1917426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:15.6970392Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:43:15.6993136Z 
2022-09-27T16:43:15.6993578Z Running tests...
2022-09-27T16:43:15.6994068Z ----------------------------------------------------------------------
2022-09-27T16:43:15.7011553Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-09-27T16:43:15.7012642Z 
2022-09-27T16:43:15.7012948Z ----------------------------------------------------------------------
2022-09-27T16:43:15.7013296Z Ran 1 test in 0.002s
2022-09-27T16:43:15.7013462Z 
2022-09-27T16:43:15.7013555Z OK (skipped=1)
2022-09-27T16:43:15.7013712Z 
2022-09-27T16:43:15.7013841Z Generating XML reports...
2022-09-27T16:43:15.7048908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164315.xml
2022-09-27T16:43:15.7415749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:15.7416392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:15.7417852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:15.7418336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:15.7469735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:15.7470185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:15.7473306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:15.7473799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:16.2713833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:16.2715313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:16.2715909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:16.2716589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:16.2902828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:16.3731489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:17.3140514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:17.3141026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:17.3144065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:17.3144682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:17.7233965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:17.7234486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:17.7236500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:17.7237003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:17.7332348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpjm4l5a
2022-09-27T16:43:17.7335007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpjm4l5a/_remote_module_non_scriptable.py
2022-09-27T16:43:17.7738898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzg3yvuco
2022-09-27T16:43:17.7739754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzg3yvuco/_remote_module_non_scriptable.py
2022-09-27T16:43:17.8414824Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:43:17.8438269Z 
2022-09-27T16:43:17.8438640Z Running tests...
2022-09-27T16:43:17.8439081Z ----------------------------------------------------------------------
2022-09-27T16:43:18.2339030Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:43:18.2361543Z 
2022-09-27T16:43:18.2361841Z Running tests...
2022-09-27T16:43:18.2362278Z ----------------------------------------------------------------------
2022-09-27T16:43:18.2380140Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-09-27T16:43:18.2380455Z 
2022-09-27T16:43:18.2380753Z ----------------------------------------------------------------------
2022-09-27T16:43:18.2381067Z Ran 1 test in 0.002s
2022-09-27T16:43:18.2381233Z 
2022-09-27T16:43:18.2381342Z OK (skipped=1)
2022-09-27T16:43:18.2381847Z 
2022-09-27T16:43:18.2381975Z Generating XML reports...
2022-09-27T16:43:18.2416992Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164318.xml
2022-09-27T16:43:18.2720455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:43:18.2721014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:43:18.8554514Z ok (6.491s)
2022-09-27T16:43:18.8554908Z 
2022-09-27T16:43:18.8555486Z ----------------------------------------------------------------------
2022-09-27T16:43:18.8555834Z Ran 1 test in 6.491s
2022-09-27T16:43:18.8556008Z 
2022-09-27T16:43:18.8556102Z OK
2022-09-27T16:43:18.8556221Z 
2022-09-27T16:43:18.8556372Z Generating XML reports...
2022-09-27T16:43:18.8609078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164312.xml
2022-09-27T16:43:19.5790851Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:19.5990839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113776
2022-09-27T16:43:19.5997918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113777
2022-09-27T16:43:20.4091194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:20.4099884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:20.4106864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:20.4112266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:20.9218028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:43:20.9240349Z 
2022-09-27T16:43:20.9240518Z Running tests...
2022-09-27T16:43:20.9241870Z ----------------------------------------------------------------------
2022-09-27T16:43:20.9260616Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-09-27T16:43:20.9260940Z 
2022-09-27T16:43:20.9261238Z ----------------------------------------------------------------------
2022-09-27T16:43:20.9261554Z Ran 1 test in 0.002s
2022-09-27T16:43:20.9261722Z 
2022-09-27T16:43:20.9261834Z OK (skipped=1)
2022-09-27T16:43:20.9261995Z 
2022-09-27T16:43:20.9262123Z Generating XML reports...
2022-09-27T16:43:20.9297369Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164320.xml
2022-09-27T16:43:21.1238360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:21.1239310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:21.1240788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:21.1241316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:21.2903052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:21.2903776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:21.2905156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:21.2905643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:21.3242545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:21.3243066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:21.3244744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:21.3245626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:21.6216643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:43:21.6239267Z 
2022-09-27T16:43:21.6239650Z Running tests...
2022-09-27T16:43:21.6240126Z ----------------------------------------------------------------------
2022-09-27T16:43:21.7968766Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:21.8240048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:21.8241080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:21.8273147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:21.8280213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:21.9260112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:22.3065610Z skip: Skipped due to small world size. (4.462s)
2022-09-27T16:43:22.3065878Z 
2022-09-27T16:43:22.3066273Z ----------------------------------------------------------------------
2022-09-27T16:43:22.3066615Z Ran 1 test in 4.463s
2022-09-27T16:43:22.3066781Z 
2022-09-27T16:43:22.3066874Z OK (skipped=1)
2022-09-27T16:43:22.3067039Z 
2022-09-27T16:43:22.3067166Z Generating XML reports...
2022-09-27T16:43:22.3102324Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164317.xml
2022-09-27T16:43:22.8916115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:22.8916635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:22.8918701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:22.8919194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:23.3348534Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:23.3516768Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.727s)
2022-09-27T16:43:23.3517341Z 
2022-09-27T16:43:23.3517614Z ----------------------------------------------------------------------
2022-09-27T16:43:23.3518213Z Ran 1 test in 1.728s
2022-09-27T16:43:23.3518400Z 
2022-09-27T16:43:23.3518514Z OK (skipped=1)
2022-09-27T16:43:23.3518671Z 
2022-09-27T16:43:23.3518797Z Generating XML reports...
2022-09-27T16:43:23.3570961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164321.xml
2022-09-27T16:43:23.3998960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:43:23.4021721Z 
2022-09-27T16:43:23.4022180Z Running tests...
2022-09-27T16:43:23.4022703Z ----------------------------------------------------------------------
2022-09-27T16:43:24.4596692Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:24.4597302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:24.4600015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:24.4600734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:25.0036304Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:43:25.0058616Z 
2022-09-27T16:43:25.0059198Z Running tests...
2022-09-27T16:43:25.0059715Z ----------------------------------------------------------------------
2022-09-27T16:43:25.0654758Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:25.0854106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113979
2022-09-27T16:43:25.0860249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113980
2022-09-27T16:43:25.5484612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:25.5485129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:25.5487485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:25.5488238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:26.1571929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:43:26.1595165Z 
2022-09-27T16:43:26.1595693Z Running tests...
2022-09-27T16:43:26.1596351Z ----------------------------------------------------------------------
2022-09-27T16:43:26.5879029Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:26.6079128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114046
2022-09-27T16:43:26.6085138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114047
2022-09-27T16:43:26.7170635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:26.7171117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:26.7172826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:26.7173311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:26.7428765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:26.7429273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:26.7431655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:26.7432137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:27.2448601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:27.2523577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:27.2525834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:27.2549974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:27.2568561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:27.2572584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:43:27.3542505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:27.3543637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:43:27.3545210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:27.3590974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:27.3612460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:43:27.3656717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:43:27.3716473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:43:27.3717844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:43:27.8375933Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:27.8558636Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.696s)
2022-09-27T16:43:27.8559936Z 
2022-09-27T16:43:27.8560901Z ----------------------------------------------------------------------
2022-09-27T16:43:27.8561504Z Ran 1 test in 1.696s
2022-09-27T16:43:27.8561828Z 
2022-09-27T16:43:27.8562030Z OK (skipped=1)
2022-09-27T16:43:27.8562321Z 
2022-09-27T16:43:27.8562536Z Generating XML reports...
2022-09-27T16:43:27.8596101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164326.xml
2022-09-27T16:43:28.2424424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:28.2425225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:28.2426247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:28.2427017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:28.2524881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:28.2525925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:28.2527131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:28.2528067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:28.7787517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:28.8157053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:28.8158653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:28.8192349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:28.8199326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:28.9170531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:29.3154613Z skip: Skipped due to small world size. (4.309s)
2022-09-27T16:43:29.3154878Z 
2022-09-27T16:43:29.3155240Z ----------------------------------------------------------------------
2022-09-27T16:43:29.3155588Z Ran 1 test in 4.310s
2022-09-27T16:43:29.3155760Z 
2022-09-27T16:43:29.3155871Z OK (skipped=1)
2022-09-27T16:43:29.3156026Z 
2022-09-27T16:43:29.3156156Z Generating XML reports...
2022-09-27T16:43:29.3203736Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164324.xml
2022-09-27T16:43:30.0636416Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:43:30.0853895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:30.0854626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:30.0858099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:30.0858590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:30.6408005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:43:30.6431764Z 
2022-09-27T16:43:30.6432253Z Running tests...
2022-09-27T16:43:30.6432781Z ----------------------------------------------------------------------
2022-09-27T16:43:31.4828501Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:31.4829137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:31.4830111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:31.4830629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:31.9742795Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:43:31.9765212Z 
2022-09-27T16:43:31.9765690Z Running tests...
2022-09-27T16:43:31.9766172Z ----------------------------------------------------------------------
2022-09-27T16:43:32.2230048Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:32.2431901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114209
2022-09-27T16:43:32.2438560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114210
2022-09-27T16:43:33.5367207Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:33.5571012Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114274
2022-09-27T16:43:33.5576439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114275
2022-09-27T16:43:33.8958027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:33.8958578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:33.8961413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:33.8961942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:33.8966807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:33.8967604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:33.8970858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:33.8971357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:34.5250542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:34.5251113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:34.5251913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:34.5252621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:34.5259131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:34.5259649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:35.1943230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:35.1944031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:35.1945390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:35.1945882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:35.2328346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:35.2329092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:35.2331829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:35.2332869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:35.7231489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:35.7410461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:35.7412141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:35.7435240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:35.7442415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:35.8423808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:35.8450149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqolqp1uk
2022-09-27T16:43:35.8452454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqolqp1uk/_remote_module_non_scriptable.py
2022-09-27T16:43:36.1638052Z skip: Skipped due to small world size. (4.187s)
2022-09-27T16:43:36.1638313Z 
2022-09-27T16:43:36.1638679Z ----------------------------------------------------------------------
2022-09-27T16:43:36.1639049Z Ran 1 test in 4.187s
2022-09-27T16:43:36.1639218Z 
2022-09-27T16:43:36.1639328Z OK (skipped=1)
2022-09-27T16:43:36.1639485Z 
2022-09-27T16:43:36.1641512Z Generating XML reports...
2022-09-27T16:43:36.1675748Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164331.xml
2022-09-27T16:43:38.1423450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:38.1423979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:38.1426057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:38.1426807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:38.6604182Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:43:38.6627593Z 
2022-09-27T16:43:38.6627860Z Running tests...
2022-09-27T16:43:38.6628298Z ----------------------------------------------------------------------
2022-09-27T16:43:40.2436576Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:40.2628050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114389
2022-09-27T16:43:40.2635383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114390
2022-09-27T16:43:41.9034605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:41.9035337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:41.9036226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:41.9036735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:41.9119926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:41.9120638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:41.9124155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:41.9124656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:42.4253866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:42.4268798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:42.4270003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:42.4355131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:42.4361456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:42.4426863Z STAGE:2022-09-27 16:43:42 114389:114389 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:42.5279586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:42.5343813Z STAGE:2022-09-27 16:43:42 114390:114390 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:44.1449258Z [1664297024.144410] [28fd6574525b:114390:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:43:44.1449803Z [1664297024.144410] [28fd6574525b:114390:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:43:44.1674566Z [1664297024.167091] [28fd6574525b:114389:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:43:44.1675087Z [1664297024.167091] [28fd6574525b:114389:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:43:44.5460602Z STAGE:2022-09-27 16:43:44 114390:114390 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:43:44 114389:114389 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:44.5461048Z 
2022-09-27T16:43:44.6699412Z STAGE:2022-09-27 16:43:44 114390:114390 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:44.6741643Z STAGE:2022-09-27 16:43:44 114389:114389 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:44.6752866Z STAGE:2022-09-27 16:43:44 114390:114390 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:43:44 114389:114389 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:44.6753303Z 
2022-09-27T16:43:45.9749609Z ok (7.312s)
2022-09-27T16:43:45.9749829Z 
2022-09-27T16:43:45.9750223Z ----------------------------------------------------------------------
2022-09-27T16:43:45.9750593Z Ran 1 test in 7.312s
2022-09-27T16:43:45.9750742Z 
2022-09-27T16:43:45.9750840Z OK
2022-09-27T16:43:45.9750981Z 
2022-09-27T16:43:45.9751128Z Generating XML reports...
2022-09-27T16:43:45.9786658Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164338.xml
2022-09-27T16:43:46.4264268Z ok (23.024s)
2022-09-27T16:43:46.4264497Z 
2022-09-27T16:43:46.4264890Z ----------------------------------------------------------------------
2022-09-27T16:43:46.4265248Z Ran 1 test in 23.024s
2022-09-27T16:43:46.4265522Z 
2022-09-27T16:43:46.4265701Z OK
2022-09-27T16:43:46.4265939Z 
2022-09-27T16:43:46.4266078Z Generating XML reports...
2022-09-27T16:43:46.4302950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164323.xml
2022-09-27T16:43:47.1694192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph754enxd
2022-09-27T16:43:47.1696585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph754enxd/_remote_module_non_scriptable.py
2022-09-27T16:43:48.2705924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:48.2706441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:48.2708709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:48.2709204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:48.2742454Z ok (17.631s)
2022-09-27T16:43:48.2742819Z 
2022-09-27T16:43:48.2743179Z ----------------------------------------------------------------------
2022-09-27T16:43:48.2743521Z Ran 1 test in 17.631s
2022-09-27T16:43:48.2743690Z 
2022-09-27T16:43:48.2743864Z OK
2022-09-27T16:43:48.2743999Z 
2022-09-27T16:43:48.2744133Z Generating XML reports...
2022-09-27T16:43:48.2780784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164330.xml
2022-09-27T16:43:48.5244861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:48.5245398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:48.5246875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:48.5247363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:48.7841291Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:43:48.7865113Z 
2022-09-27T16:43:48.7865601Z Running tests...
2022-09-27T16:43:48.7866094Z ----------------------------------------------------------------------
2022-09-27T16:43:49.0107775Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:43:49.0130047Z 
2022-09-27T16:43:49.0130328Z Running tests...
2022-09-27T16:43:49.0130796Z ----------------------------------------------------------------------
2022-09-27T16:43:50.3776838Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:50.3777474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:50.3779619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:50.3780409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:50.4391755Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:50.4588621Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114576
2022-09-27T16:43:50.4593877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114577
2022-09-27T16:43:50.6194860Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:50.6388589Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114579
2022-09-27T16:43:50.6394795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114580
2022-09-27T16:43:50.8732609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:43:50.8755094Z 
2022-09-27T16:43:50.8755352Z Running tests...
2022-09-27T16:43:50.8755770Z ----------------------------------------------------------------------
2022-09-27T16:43:52.2163744Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:52.2164339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:52.2166656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:52.2167152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:52.2687183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:52.2687845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:52.2693871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:52.2694811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:52.3940969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:52.3941457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:52.3943721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:52.3944222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:52.4330418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:52.4330876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:52.4333995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:52.4334484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:52.5797579Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:52.5995451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114711
2022-09-27T16:43:52.6001627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114712
2022-09-27T16:43:52.7380756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:52.7717399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:52.7718216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:52.7785839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:52.7792927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:52.7860084Z STAGE:2022-09-27 16:43:52 114577:114577 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:52.8731219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:52.8796274Z STAGE:2022-09-27 16:43:52 114576:114576 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:52.9382176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:52.9610329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:52.9611107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:52.9686658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:52.9694272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:52.9697548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:43:53.0624693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:53.0628248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:43:53.0629292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:53.0715294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:53.0843248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:43:53.0843789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:43:53.0844500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:43:53.0845217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:43:54.2439462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:54.2440025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:54.2441611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:54.2442125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:54.2966267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:54.2966901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:54.2968599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:54.2969341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:54.6921579Z [1664297034.691515] [28fd6574525b:114576:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:43:54.7175913Z [1664297034.691515] [28fd6574525b:114576:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:43:54.7176686Z [1664297034.717112] [28fd6574525b:114577:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:43:54.7177210Z [1664297034.717112] [28fd6574525b:114577:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:43:54.8672907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:43:54.8673965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:43:54.8675027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:54.8675752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:43:54.8681929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:43:54.8683017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:43:54.8792963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:43:54.8794066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:43:54.8794843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:54.8795549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:43:55.1098087Z STAGE:2022-09-27 16:43:55 114577:114577 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:55.1111177Z STAGE:2022-09-27 16:43:55 114576:114576 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:55.2408238Z STAGE:2022-09-27 16:43:55 114577:114577 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:55.2425243Z STAGE:2022-09-27 16:43:55 114576:114576 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:43:55.2435474Z STAGE:2022-09-27 16:43:55 114577:114577 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:55.2436581Z STAGE:2022-09-27 16:43:55 114576:114576 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:43:55.4072218Z ok (4.531s)
2022-09-27T16:43:55.4072594Z 
2022-09-27T16:43:55.4073356Z ----------------------------------------------------------------------
2022-09-27T16:43:55.4073891Z Ran 1 test in 4.532s
2022-09-27T16:43:55.4074065Z 
2022-09-27T16:43:55.4074178Z OK
2022-09-27T16:43:55.4074314Z 
2022-09-27T16:43:55.4074456Z Generating XML reports...
2022-09-27T16:43:55.4109470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164350.xml
2022-09-27T16:43:55.8190940Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:43:55.8191968Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:43:56.2717270Z ok (7.485s)
2022-09-27T16:43:56.2717462Z 
2022-09-27T16:43:56.2718169Z ----------------------------------------------------------------------
2022-09-27T16:43:56.2718873Z Ran 1 test in 7.485s
2022-09-27T16:43:56.2719124Z 
2022-09-27T16:43:56.2719223Z OK
2022-09-27T16:43:56.2719361Z 
2022-09-27T16:43:56.2719494Z Generating XML reports...
2022-09-27T16:43:56.2755357Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164348.xml
2022-09-27T16:43:57.6025417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:57.6025932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:57.6028067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:57.6028551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:58.1033686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:43:58.1057520Z 
2022-09-27T16:43:58.1057746Z Running tests...
2022-09-27T16:43:58.1058205Z ----------------------------------------------------------------------
2022-09-27T16:43:58.3634064Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:43:58.3634979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:43:58.3636082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:43:58.3636625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:43:58.8542142Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:43:58.8563992Z 
2022-09-27T16:43:58.8564390Z Running tests...
2022-09-27T16:43:58.8564872Z ----------------------------------------------------------------------
2022-09-27T16:43:58.8585436Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-09-27T16:43:58.8585774Z 
2022-09-27T16:43:58.8586058Z ----------------------------------------------------------------------
2022-09-27T16:43:58.8586393Z Ran 1 test in 0.002s
2022-09-27T16:43:58.8586561Z 
2022-09-27T16:43:58.8586672Z OK (skipped=1)
2022-09-27T16:43:58.8586855Z 
2022-09-27T16:43:58.8586969Z Generating XML reports...
2022-09-27T16:43:58.8621398Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164358.xml
2022-09-27T16:43:59.7083845Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:43:59.7284992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114899
2022-09-27T16:43:59.7292634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114900
2022-09-27T16:44:00.8047773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:00.8048427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:00.8049962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:00.8050438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:01.2966119Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:01.2989866Z 
2022-09-27T16:44:01.2990311Z Running tests...
2022-09-27T16:44:01.2990794Z ----------------------------------------------------------------------
2022-09-27T16:44:01.3757850Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:01.3758895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:01.3760433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:01.3760937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:01.3873862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:01.3874367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:01.3877660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:01.3878196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:01.9393155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:01.9394217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:01.9395715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:01.9397119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:01.9401865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:01.9403164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:01.9509867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:44:01.9510808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:44:01.9512142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:44:01.9612415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:44:02.4358455Z ok (4.330s)
2022-09-27T16:44:02.4358851Z 
2022-09-27T16:44:02.4359522Z ----------------------------------------------------------------------
2022-09-27T16:44:02.4360457Z Ran 1 test in 4.330s
2022-09-27T16:44:02.4360817Z 
2022-09-27T16:44:02.4360997Z OK
2022-09-27T16:44:02.4361260Z 
2022-09-27T16:44:02.4362293Z Generating XML reports...
2022-09-27T16:44:02.4395834Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164358.xml
2022-09-27T16:44:02.8756596Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:02.8949815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115014
2022-09-27T16:44:02.8955689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115015
2022-09-27T16:44:04.5319357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:04.5319867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:04.5321510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:04.5322020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:04.5631003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:04.5631461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:04.5634406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:04.5634911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:04.5639198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:04.5639648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:04.5643531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:04.5644025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:05.0393801Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:44:05.0415955Z 
2022-09-27T16:44:05.0416400Z Running tests...
2022-09-27T16:44:05.0416947Z ----------------------------------------------------------------------
2022-09-27T16:44:05.0732135Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:05.0754089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:05.0754830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:05.0833789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:05.0840151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:05.0914258Z STAGE:2022-09-27 16:44:05 115015:115015 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:05.1768769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:05.1833392Z STAGE:2022-09-27 16:44:05 115014:115014 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:06.6854992Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:06.7054062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115124
2022-09-27T16:44:06.7059010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115125
2022-09-27T16:44:06.7945593Z [1664297046.794020] [28fd6574525b:115015:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:44:06.7946128Z [1664297046.794020] [28fd6574525b:115015:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:44:06.8133054Z [1664297046.812842] [28fd6574525b:115014:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:44:06.8133598Z [1664297046.812842] [28fd6574525b:115014:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:44:07.2241135Z STAGE:2022-09-27 16:44:07 115014:115014 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:44:07 115015:115015 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:07.2241920Z 
2022-09-27T16:44:07.3727208Z STAGE:2022-09-27 16:44:07 115015:115015 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:07.3962147Z STAGE:2022-09-27 16:44:07 115014:115014 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:07.3981650Z STAGE:2022-09-27 16:44:07 115014:115014 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:07.4060937Z STAGE:2022-09-27 16:44:07 115015:115015 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:08.3463434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:08.3463954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:08.3465905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:08.3466387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:08.3876813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:08.3877315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:08.3878743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:08.3879237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:08.6074840Z ok (7.308s)
2022-09-27T16:44:08.6075062Z 
2022-09-27T16:44:08.6075470Z ----------------------------------------------------------------------
2022-09-27T16:44:08.6075818Z Ran 1 test in 7.308s
2022-09-27T16:44:08.6075988Z 
2022-09-27T16:44:08.6076066Z OK
2022-09-27T16:44:08.6076208Z 
2022-09-27T16:44:08.6076352Z Generating XML reports...
2022-09-27T16:44:08.6112763Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164401.xml
2022-09-27T16:44:08.9186854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:08.9187417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:08.9188220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:08.9188915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:08.9295376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:09.0201764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:10.3790465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxuauamh9
2022-09-27T16:44:10.3791838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxuauamh9/_remote_module_non_scriptable.py
2022-09-27T16:44:10.6767043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:10.6768192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:10.6769592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:10.6770723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:11.2173249Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:11.2196902Z 
2022-09-27T16:44:11.2197353Z Running tests...
2022-09-27T16:44:11.2198221Z ----------------------------------------------------------------------
2022-09-27T16:44:11.2228742Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-09-27T16:44:11.2229407Z 
2022-09-27T16:44:11.2229989Z ----------------------------------------------------------------------
2022-09-27T16:44:11.2230641Z Ran 1 test in 0.003s
2022-09-27T16:44:11.2230983Z 
2022-09-27T16:44:11.2231171Z OK (skipped=1)
2022-09-27T16:44:11.2231487Z 
2022-09-27T16:44:11.2231742Z Generating XML reports...
2022-09-27T16:44:11.2267383Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164411.xml
2022-09-27T16:44:11.4791515Z ok (22.466s)
2022-09-27T16:44:11.4791892Z 
2022-09-27T16:44:11.4792523Z ----------------------------------------------------------------------
2022-09-27T16:44:11.4793154Z Ran 1 test in 22.466s
2022-09-27T16:44:11.4793458Z 
2022-09-27T16:44:11.4793600Z OK
2022-09-27T16:44:11.4793851Z 
2022-09-27T16:44:11.4794091Z Generating XML reports...
2022-09-27T16:44:11.4828885Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164349.xml
2022-09-27T16:44:12.1030036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3w8rfcw
2022-09-27T16:44:12.1031215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3w8rfcw/_remote_module_non_scriptable.py
2022-09-27T16:44:12.5621488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:44:12.5622057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:44:12.5705262Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:44:12.5706881Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:44:13.1206847Z ok (8.079s)
2022-09-27T16:44:13.1207064Z 
2022-09-27T16:44:13.1207776Z ----------------------------------------------------------------------
2022-09-27T16:44:13.1208129Z Ran 1 test in 8.079s
2022-09-27T16:44:13.1208301Z 
2022-09-27T16:44:13.1208396Z OK
2022-09-27T16:44:13.1208547Z 
2022-09-27T16:44:13.1208684Z Generating XML reports...
2022-09-27T16:44:13.1243231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164405.xml
2022-09-27T16:44:13.2249952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:13.2250483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:13.2253184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:13.2253678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:13.6569844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:13.6570375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:13.6571766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:13.6572545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:13.7150489Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:13.7173881Z 
2022-09-27T16:44:13.7174102Z Running tests...
2022-09-27T16:44:13.7174540Z ----------------------------------------------------------------------
2022-09-27T16:44:14.1635480Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:14.1658554Z 
2022-09-27T16:44:14.1658892Z Running tests...
2022-09-27T16:44:14.1659347Z ----------------------------------------------------------------------
2022-09-27T16:44:14.1680145Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:44:14.1680723Z 
2022-09-27T16:44:14.1681001Z ----------------------------------------------------------------------
2022-09-27T16:44:14.1681318Z Ran 1 test in 0.002s
2022-09-27T16:44:14.1681487Z 
2022-09-27T16:44:14.1681599Z OK (skipped=1)
2022-09-27T16:44:14.1681758Z 
2022-09-27T16:44:14.1681884Z Generating XML reports...
2022-09-27T16:44:14.1737682Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164414.xml
2022-09-27T16:44:15.1831177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:15.1832822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:15.1835160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:15.1836257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:15.3510401Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:15.3704354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115375
2022-09-27T16:44:15.3710948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115376
2022-09-27T16:44:15.7359324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:44:15.7383183Z 
2022-09-27T16:44:15.7383489Z Running tests...
2022-09-27T16:44:15.7383940Z ----------------------------------------------------------------------
2022-09-27T16:44:16.2854674Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:16.2855292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:16.2857785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:16.2858312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:16.7919295Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:16.7942573Z 
2022-09-27T16:44:16.7943059Z Running tests...
2022-09-27T16:44:16.7943544Z ----------------------------------------------------------------------
2022-09-27T16:44:16.7973443Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-09-27T16:44:16.7973953Z 
2022-09-27T16:44:16.7974222Z ----------------------------------------------------------------------
2022-09-27T16:44:16.7974793Z Ran 1 test in 0.003s
2022-09-27T16:44:16.7975007Z 
2022-09-27T16:44:16.7975130Z OK (skipped=1)
2022-09-27T16:44:16.7975297Z 
2022-09-27T16:44:16.7975438Z Generating XML reports...
2022-09-27T16:44:16.8015706Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164416.xml
2022-09-27T16:44:17.0357856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:17.0358364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:17.0359551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:17.0360049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:17.1058757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:17.1059266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:17.1061172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:17.1061682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:17.3639257Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:17.3839551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115445
2022-09-27T16:44:17.3846691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115446
2022-09-27T16:44:17.5476434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:17.6169714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:17.6170556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:17.6186141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:17.6193970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:17.6260811Z STAGE:2022-09-27 16:44:17 115376:115376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:17.7186843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:17.7289765Z STAGE:2022-09-27 16:44:17 115375:115375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:18.7909332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:18.7909845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:18.7912647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:18.7913147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:19.0800765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:19.0801310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:19.0802242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:19.0802749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:19.1023906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:19.1024404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:19.1026387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:19.1026880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:19.3102259Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:19.3126468Z 
2022-09-27T16:44:19.3126608Z Running tests...
2022-09-27T16:44:19.3127360Z ----------------------------------------------------------------------
2022-09-27T16:44:19.3148175Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:44:19.3148524Z 
2022-09-27T16:44:19.3148795Z ----------------------------------------------------------------------
2022-09-27T16:44:19.3149120Z Ran 1 test in 0.002s
2022-09-27T16:44:19.3149292Z 
2022-09-27T16:44:19.3149411Z OK (skipped=1)
2022-09-27T16:44:19.3149574Z 
2022-09-27T16:44:19.3149705Z Generating XML reports...
2022-09-27T16:44:19.3186088Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164419.xml
2022-09-27T16:44:19.3894291Z [1664297059.388954] [28fd6574525b:115375:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:44:19.3894863Z [1664297059.388954] [28fd6574525b:115375:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:44:19.4039203Z [1664297059.403481] [28fd6574525b:115376:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:44:19.4039911Z [1664297059.403481] [28fd6574525b:115376:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:44:19.6398155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:19.6399258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:19.6401398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:19.6402879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:19.6407202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:19.6408198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:19.8224514Z STAGE:2022-09-27 16:44:19 115376:115376 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:44:19 115375:115375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:19.8224963Z 
2022-09-27T16:44:19.9473385Z STAGE:2022-09-27 16:44:19 115376:115376 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:19.9481577Z STAGE:2022-09-27 16:44:19 115375:115375 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:19.9491991Z STAGE:2022-09-27 16:44:19 115376:115376 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:19.9493277Z STAGE:2022-09-27 16:44:19 115375:115375 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:21.0285188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrycq39g
2022-09-27T16:44:21.0286083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrycq39g/_remote_module_non_scriptable.py
2022-09-27T16:44:21.0376902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9q6e3k8
2022-09-27T16:44:21.0379670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9q6e3k8/_remote_module_non_scriptable.py
2022-09-27T16:44:21.1850533Z ok (7.467s)
2022-09-27T16:44:21.1850759Z 
2022-09-27T16:44:21.1851151Z ----------------------------------------------------------------------
2022-09-27T16:44:21.1851483Z Ran 1 test in 7.468s
2022-09-27T16:44:21.1851649Z 
2022-09-27T16:44:21.1851754Z OK
2022-09-27T16:44:21.1851893Z 
2022-09-27T16:44:21.1852033Z Generating XML reports...
2022-09-27T16:44:21.1887266Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164413.xml
2022-09-27T16:44:21.3196670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:21.3197183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:21.3198871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:21.3199369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:21.5588186Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:44:21.5799188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:44:21.5799750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:44:21.8647202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:21.8670750Z 
2022-09-27T16:44:21.8671270Z Running tests...
2022-09-27T16:44:21.8671796Z ----------------------------------------------------------------------
2022-09-27T16:44:21.8692943Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:44:21.8693360Z 
2022-09-27T16:44:21.8693708Z ----------------------------------------------------------------------
2022-09-27T16:44:21.8694063Z Ran 1 test in 0.002s
2022-09-27T16:44:21.8694243Z 
2022-09-27T16:44:21.8694355Z OK (skipped=1)
2022-09-27T16:44:21.8694518Z 
2022-09-27T16:44:21.8694641Z Generating XML reports...
2022-09-27T16:44:21.8729530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164421.xml
2022-09-27T16:44:22.0957994Z ok (6.357s)
2022-09-27T16:44:22.0958225Z 
2022-09-27T16:44:22.0958614Z ----------------------------------------------------------------------
2022-09-27T16:44:22.0958944Z Ran 1 test in 6.357s
2022-09-27T16:44:22.0959103Z 
2022-09-27T16:44:22.0959200Z OK
2022-09-27T16:44:22.0959341Z 
2022-09-27T16:44:22.0959476Z Generating XML reports...
2022-09-27T16:44:22.0994578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164415.xml
2022-09-27T16:44:23.5461560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:23.5462551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:23.5464148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:23.5465144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:23.9058867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:23.9059414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:23.9061629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:23.9062129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:24.0362584Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:24.0385896Z 
2022-09-27T16:44:24.0386459Z Running tests...
2022-09-27T16:44:24.0386960Z ----------------------------------------------------------------------
2022-09-27T16:44:24.0408590Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:44:24.0409171Z 
2022-09-27T16:44:24.0409473Z ----------------------------------------------------------------------
2022-09-27T16:44:24.0410349Z Ran 1 test in 0.002s
2022-09-27T16:44:24.0410502Z 
2022-09-27T16:44:24.0410613Z OK (skipped=1)
2022-09-27T16:44:24.0410772Z 
2022-09-27T16:44:24.0410900Z Generating XML reports...
2022-09-27T16:44:24.0445459Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164424.xml
2022-09-27T16:44:24.2650634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:24.2651437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:24.2653655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:24.2654170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:24.4094079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:24.4116063Z 
2022-09-27T16:44:24.4116570Z Running tests...
2022-09-27T16:44:24.4117064Z ----------------------------------------------------------------------
2022-09-27T16:44:24.4138272Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:44:24.4138822Z 
2022-09-27T16:44:24.4139179Z ----------------------------------------------------------------------
2022-09-27T16:44:24.4139692Z Ran 1 test in 0.002s
2022-09-27T16:44:24.4139855Z 
2022-09-27T16:44:24.4139969Z OK (skipped=1)
2022-09-27T16:44:24.4140129Z 
2022-09-27T16:44:24.4140258Z Generating XML reports...
2022-09-27T16:44:24.4174772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164424.xml
2022-09-27T16:44:24.7673355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:44:24.7696764Z 
2022-09-27T16:44:24.7696914Z Running tests...
2022-09-27T16:44:24.7697608Z ----------------------------------------------------------------------
2022-09-27T16:44:26.0262294Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:26.0262993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:26.0264632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:26.0265132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:26.3968588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:26.3969390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:26.3970841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:26.3971326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:26.4276542Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:26.4472732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115772
2022-09-27T16:44:26.4479091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115773
2022-09-27T16:44:26.5172877Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:26.5195431Z 
2022-09-27T16:44:26.5195850Z Running tests...
2022-09-27T16:44:26.5196338Z ----------------------------------------------------------------------
2022-09-27T16:44:26.5221438Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:44:26.5221872Z 
2022-09-27T16:44:26.5222470Z ----------------------------------------------------------------------
2022-09-27T16:44:26.5223144Z Ran 1 test in 0.002s
2022-09-27T16:44:26.5223709Z 
2022-09-27T16:44:26.5223823Z OK (skipped=1)
2022-09-27T16:44:26.5223961Z 
2022-09-27T16:44:26.5224093Z Generating XML reports...
2022-09-27T16:44:26.5254803Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164426.xml
2022-09-27T16:44:26.8924675Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:26.8946879Z 
2022-09-27T16:44:26.8947353Z Running tests...
2022-09-27T16:44:26.8948311Z ----------------------------------------------------------------------
2022-09-27T16:44:28.1965497Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:28.1966644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:28.1967814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:28.1968781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:28.2139868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:28.2140850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:28.2142786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:28.2143757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:28.5311561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:28.5312610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:28.5314311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:28.5315296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:28.5744908Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:28.5941342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115873
2022-09-27T16:44:28.5947421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115874
2022-09-27T16:44:28.7551264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:28.7552335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:28.7553824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:28.7555498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:28.7559376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:28.7561168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:29.0447394Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:29.0471660Z 
2022-09-27T16:44:29.0472060Z Running tests...
2022-09-27T16:44:29.0472939Z ----------------------------------------------------------------------
2022-09-27T16:44:29.1547162Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... ok (4.385s)
2022-09-27T16:44:29.1547765Z 
2022-09-27T16:44:29.1548405Z ----------------------------------------------------------------------
2022-09-27T16:44:29.1548989Z Ran 1 test in 4.385s
2022-09-27T16:44:29.1549282Z 
2022-09-27T16:44:29.1549437Z OK
2022-09-27T16:44:29.1549683Z 
2022-09-27T16:44:29.1549915Z Generating XML reports...
2022-09-27T16:44:29.1584145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164424.xml
2022-09-27T16:44:30.2428638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:30.2429161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:30.2431861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:30.2432346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:30.2611835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:30.2612327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:30.2614623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:30.2615097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:30.7481776Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:30.7675125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115981
2022-09-27T16:44:30.7681612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115982
2022-09-27T16:44:30.8046167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:30.8076014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:30.8077545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:30.8147753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:30.8154051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:30.9092527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:31.2779311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:31.2780299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:31.2781522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:31.2782416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:31.8827459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:44:31.8848831Z 
2022-09-27T16:44:31.8849357Z Running tests...
2022-09-27T16:44:31.8849864Z ----------------------------------------------------------------------
2022-09-27T16:44:32.3983689Z   test_gather (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:32.3984787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:32.3985895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:32.3986721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:32.4632840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:32.4633807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:32.4634902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:32.4635778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:32.9046924Z ok (6.010s)
2022-09-27T16:44:32.9047156Z 
2022-09-27T16:44:32.9047567Z ----------------------------------------------------------------------
2022-09-27T16:44:32.9048200Z Ran 1 test in 6.010s
2022-09-27T16:44:32.9048369Z 
2022-09-27T16:44:32.9048466Z OK
2022-09-27T16:44:32.9048603Z 
2022-09-27T16:44:32.9048739Z Generating XML reports...
2022-09-27T16:44:32.9083249Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164426.xml
2022-09-27T16:44:32.9251790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:32.9991996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:32.9992840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:33.0063963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:33.0070975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:33.0138352Z STAGE:2022-09-27 16:44:33 115981:115981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:33.1006288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:33.1074703Z STAGE:2022-09-27 16:44:33 115982:115982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:33.5361982Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:33.5561798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116076
2022-09-27T16:44:33.5567695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116077
2022-09-27T16:44:34.9272618Z [1664297074.926648] [28fd6574525b:115982:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:44:34.9273565Z [1664297074.926648] [28fd6574525b:115982:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:44:34.9276640Z [1664297074.927329] [28fd6574525b:115981:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:44:34.9277569Z [1664297074.927329] [28fd6574525b:115981:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:44:35.1570836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:35.1571848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:35.1573055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:35.1573995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:35.3105137Z STAGE:2022-09-27 16:44:35 115981:115981 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:44:35 115982:115982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:35.3106021Z 
2022-09-27T16:44:35.3106798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:35.3107652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:35.3110958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:35.3111909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:35.3806596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:35.3807496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:35.3808861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:35.3809699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:35.4416916Z STAGE:2022-09-27 16:44:35 115982:115982 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:35.4748191Z STAGE:2022-09-27 16:44:35 115981:115981 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:35.4758257Z STAGE:2022-09-27 16:44:35 115981:115981 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:35.4759457Z STAGE:2022-09-27 16:44:35 115982:115982 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:35.6841344Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:35.6865547Z 
2022-09-27T16:44:35.6866042Z Running tests...
2022-09-27T16:44:35.6866562Z ----------------------------------------------------------------------
2022-09-27T16:44:35.9275321Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:35.9275979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:35.9276756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:35.9277452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:35.9384060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:35.9450893Z STAGE:2022-09-27 16:44:35 116077:116077 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:36.0290130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:36.0356563Z STAGE:2022-09-27 16:44:36 116076:116076 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:36.0372549Z STAGE:2022-09-27 16:44:36 116077:116077 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:36.0373171Z STAGE:2022-09-27 16:44:36 116076:116076 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:36.0385465Z STAGE:2022-09-27 16:44:36 116077:116077 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:36.0395303Z STAGE:2022-09-27 16:44:36 116076:116076 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:36.0408166Z STAGE:2022-09-27 16:44:36 116076:116076 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:36.0408936Z STAGE:2022-09-27 16:44:36 116077:116077 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:36.4640315Z ok (4.579s)
2022-09-27T16:44:36.4640790Z 
2022-09-27T16:44:36.4641463Z ----------------------------------------------------------------------
2022-09-27T16:44:36.4641811Z Ran 1 test in 4.579s
2022-09-27T16:44:36.4641987Z 
2022-09-27T16:44:36.4642086Z OK
2022-09-27T16:44:36.4642225Z 
2022-09-27T16:44:36.4642362Z Generating XML reports...
2022-09-27T16:44:36.4678392Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164431.xml
2022-09-27T16:44:36.5812167Z ok (7.534s)
2022-09-27T16:44:36.5812384Z 
2022-09-27T16:44:36.5812776Z ----------------------------------------------------------------------
2022-09-27T16:44:36.5813118Z Ran 1 test in 7.534s
2022-09-27T16:44:36.5813296Z 
2022-09-27T16:44:36.5813373Z OK
2022-09-27T16:44:36.5813512Z 
2022-09-27T16:44:36.5813647Z Generating XML reports...
2022-09-27T16:44:36.5938995Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164429.xml
2022-09-27T16:44:37.3869884Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:37.4070984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116195
2022-09-27T16:44:37.4077597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116196
2022-09-27T16:44:38.7871674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:38.7872702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:38.7874165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:38.7874848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:38.8414385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:38.8414878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:38.8418108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:38.8418611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:39.1566336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:39.1566860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:39.1568121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:39.1568592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:39.2437571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:39.2438061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:39.2439995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:39.2440695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:39.2873956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:44:39.2898161Z 
2022-09-27T16:44:39.2898372Z Running tests...
2022-09-27T16:44:39.2898805Z ----------------------------------------------------------------------
2022-09-27T16:44:39.3332402Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:39.3355713Z 
2022-09-27T16:44:39.3355979Z Running tests...
2022-09-27T16:44:39.3356417Z ----------------------------------------------------------------------
2022-09-27T16:44:39.3376002Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:44:39.3376360Z 
2022-09-27T16:44:39.3376680Z ----------------------------------------------------------------------
2022-09-27T16:44:39.3377025Z Ran 1 test in 0.002s
2022-09-27T16:44:39.3377478Z 
2022-09-27T16:44:39.3377609Z OK (skipped=1)
2022-09-27T16:44:39.3377771Z 
2022-09-27T16:44:39.3377901Z Generating XML reports...
2022-09-27T16:44:39.3412205Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164439.xml
2022-09-27T16:44:39.6665196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:39.7588469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:39.7589308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:39.7677433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:39.7684879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:39.8602390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:41.0140083Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:41.0347580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116367
2022-09-27T16:44:41.0353510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116368
2022-09-27T16:44:41.3189738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:41.3190252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:41.3192463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:41.3192963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:41.8177872Z ok (6.131s)
2022-09-27T16:44:41.8178087Z 
2022-09-27T16:44:41.8178510Z ----------------------------------------------------------------------
2022-09-27T16:44:41.8178852Z Ran 1 test in 6.131s
2022-09-27T16:44:41.8179021Z 
2022-09-27T16:44:41.8179099Z OK
2022-09-27T16:44:41.8179238Z 
2022-09-27T16:44:41.8179371Z Generating XML reports...
2022-09-27T16:44:41.8215198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164435.xml
2022-09-27T16:44:41.8867414Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:41.8891983Z 
2022-09-27T16:44:41.8892238Z Running tests...
2022-09-27T16:44:41.8892692Z ----------------------------------------------------------------------
2022-09-27T16:44:41.8917184Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:44:41.8918052Z 
2022-09-27T16:44:41.8918504Z ----------------------------------------------------------------------
2022-09-27T16:44:41.8919059Z Ran 1 test in 0.003s
2022-09-27T16:44:41.8919284Z 
2022-09-27T16:44:41.8919420Z OK (skipped=1)
2022-09-27T16:44:41.8919585Z 
2022-09-27T16:44:41.8919695Z Generating XML reports...
2022-09-27T16:44:41.8954617Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164441.xml
2022-09-27T16:44:42.7062351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:42.7063355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:42.7064521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:42.7065473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:42.7248256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:42.7249197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:42.7252590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:42.7253562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:43.3875497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:43.3876074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:43.3876879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:43.3877576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:43.3884631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:43.3885151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:43.7418776Z ok (4.452s)
2022-09-27T16:44:43.7418994Z 
2022-09-27T16:44:43.7419369Z ----------------------------------------------------------------------
2022-09-27T16:44:43.7420038Z Ran 1 test in 4.452s
2022-09-27T16:44:43.7420208Z 
2022-09-27T16:44:43.7420304Z OK
2022-09-27T16:44:43.7420441Z 
2022-09-27T16:44:43.7420584Z Generating XML reports...
2022-09-27T16:44:43.7456815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164439.xml
2022-09-27T16:44:43.9214052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:43.9214601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:43.9216730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:43.9217243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:43.9567152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:43.9567677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:43.9570611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:43.9571098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:44.4441961Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:44.4463990Z 
2022-09-27T16:44:44.4464436Z Running tests...
2022-09-27T16:44:44.4464971Z ----------------------------------------------------------------------
2022-09-27T16:44:44.4647493Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:44.4672430Z 
2022-09-27T16:44:44.4672810Z Running tests...
2022-09-27T16:44:44.4673232Z ----------------------------------------------------------------------
2022-09-27T16:44:44.4691504Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:44:44.4691846Z 
2022-09-27T16:44:44.4692128Z ----------------------------------------------------------------------
2022-09-27T16:44:44.4692460Z Ran 1 test in 0.002s
2022-09-27T16:44:44.4692608Z 
2022-09-27T16:44:44.4692731Z OK (skipped=1)
2022-09-27T16:44:44.4692886Z 
2022-09-27T16:44:44.4693023Z Generating XML reports...
2022-09-27T16:44:44.4728013Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164444.xml
2022-09-27T16:44:45.8867308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:45.8867832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:45.8869584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:45.8870112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:46.0583595Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:46.0777456Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116582
2022-09-27T16:44:46.0783767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116583
2022-09-27T16:44:46.3862509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:44:46.3886481Z 
2022-09-27T16:44:46.3886679Z Running tests...
2022-09-27T16:44:46.3887123Z ----------------------------------------------------------------------
2022-09-27T16:44:46.3905802Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:44:46.3906134Z 
2022-09-27T16:44:46.3906526Z ----------------------------------------------------------------------
2022-09-27T16:44:46.3907115Z Ran 1 test in 0.002s
2022-09-27T16:44:46.3907288Z 
2022-09-27T16:44:46.3907405Z OK (skipped=1)
2022-09-27T16:44:46.3907861Z 
2022-09-27T16:44:46.3907991Z Generating XML reports...
2022-09-27T16:44:46.3942177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164446.xml
2022-09-27T16:44:46.4834295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:46.4834810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:46.4836969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:46.4837466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:46.9857716Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:46.9880631Z 
2022-09-27T16:44:46.9881208Z Running tests...
2022-09-27T16:44:46.9881676Z ----------------------------------------------------------------------
2022-09-27T16:44:46.9902139Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:44:46.9902497Z 
2022-09-27T16:44:46.9903062Z ----------------------------------------------------------------------
2022-09-27T16:44:46.9903416Z Ran 1 test in 0.002s
2022-09-27T16:44:46.9903586Z 
2022-09-27T16:44:46.9903697Z OK (skipped=1)
2022-09-27T16:44:46.9903837Z 
2022-09-27T16:44:46.9905300Z Generating XML reports...
2022-09-27T16:44:46.9939411Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164446.xml
2022-09-27T16:44:47.7436073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:47.7436758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:47.7437588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:47.7438062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:47.7438664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:47.7439125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:47.7439725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:47.7440491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:48.3154091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:48.3169777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:48.3170834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:48.3254869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:48.3261576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:48.4184459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:48.4605942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:48.4606590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:48.4608185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:48.4608675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:48.9807293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:44:48.9829025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:48.9829815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:48.9830027Z 
2022-09-27T16:44:48.9830139Z Running tests...
2022-09-27T16:44:48.9830561Z ----------------------------------------------------------------------
2022-09-27T16:44:48.9834583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:48.9835066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:49.5092783Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:49.5116915Z 
2022-09-27T16:44:49.5117195Z Running tests...
2022-09-27T16:44:49.5117626Z ----------------------------------------------------------------------
2022-09-27T16:44:49.5138503Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:44:49.5139019Z 
2022-09-27T16:44:49.5139389Z ----------------------------------------------------------------------
2022-09-27T16:44:49.5139952Z Ran 1 test in 0.002s
2022-09-27T16:44:49.5140107Z 
2022-09-27T16:44:49.5140224Z OK (skipped=1)
2022-09-27T16:44:49.5140381Z 
2022-09-27T16:44:49.5140511Z Generating XML reports...
2022-09-27T16:44:49.5177978Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164449.xml
2022-09-27T16:44:50.6251618Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:50.6718871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116760
2022-09-27T16:44:50.6724641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116761
2022-09-27T16:44:51.4903012Z ok (7.044s)
2022-09-27T16:44:51.4903394Z 
2022-09-27T16:44:51.4903892Z ----------------------------------------------------------------------
2022-09-27T16:44:51.4904261Z Ran 1 test in 7.044s
2022-09-27T16:44:51.4904432Z 
2022-09-27T16:44:51.4904524Z OK
2022-09-27T16:44:51.4904648Z 
2022-09-27T16:44:51.4904781Z Generating XML reports...
2022-09-27T16:44:51.4939181Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164444.xml
2022-09-27T16:44:51.5980163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:51.5980656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:51.5983023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:51.5983865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:52.1102562Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:52.1126614Z 
2022-09-27T16:44:52.1127129Z Running tests...
2022-09-27T16:44:52.1127645Z ----------------------------------------------------------------------
2022-09-27T16:44:52.1146262Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:44:52.1147471Z 
2022-09-27T16:44:52.1147758Z ----------------------------------------------------------------------
2022-09-27T16:44:52.1148108Z Ran 1 test in 0.002s
2022-09-27T16:44:52.1148279Z 
2022-09-27T16:44:52.1148392Z OK (skipped=1)
2022-09-27T16:44:52.1148550Z 
2022-09-27T16:44:52.1148679Z Generating XML reports...
2022-09-27T16:44:52.1184977Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164452.xml
2022-09-27T16:44:52.3032393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:52.3032968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:52.3034412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:52.3035186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:52.3281126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:52.3281661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:52.3284618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:52.3285343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:52.9084644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:52.9085470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:52.9086255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:52.9086962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:52.9092854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:52.9093687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:52.9201311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:44:52.9202037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:44:52.9202721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:44:52.9203421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:44:52.9267236Z STAGE:2022-09-27 16:44:52 116760:116760 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:52.9362334Z STAGE:2022-09-27 16:44:52 116761:116761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:52.9375700Z STAGE:2022-09-27 16:44:52 116761:116761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:52.9376689Z STAGE:2022-09-27 16:44:52 116760:116760 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:52.9388347Z STAGE:2022-09-27 16:44:52 116761:116761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:52.9398962Z STAGE:2022-09-27 16:44:52 116760:116760 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:44:52.9411430Z STAGE:2022-09-27 16:44:52 116760:116760 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:52.9412286Z STAGE:2022-09-27 16:44:52 116761:116761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:44:53.4795512Z ok (4.496s)
2022-09-27T16:44:53.4795904Z 
2022-09-27T16:44:53.4796570Z ----------------------------------------------------------------------
2022-09-27T16:44:53.4797183Z Ran 1 test in 4.496s
2022-09-27T16:44:53.4797503Z 
2022-09-27T16:44:53.4797676Z OK
2022-09-27T16:44:53.4797922Z 
2022-09-27T16:44:53.4798158Z Generating XML reports...
2022-09-27T16:44:53.4831826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164448.xml
2022-09-27T16:44:53.6068494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:53.6069484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:53.6071835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:53.6072765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:54.1089076Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:44:54.1113285Z 
2022-09-27T16:44:54.1113752Z Running tests...
2022-09-27T16:44:54.1114397Z ----------------------------------------------------------------------
2022-09-27T16:44:54.1433929Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:54.1434945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:54.1437976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:54.1438832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:54.7067473Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:54.7090747Z 
2022-09-27T16:44:54.7091066Z Running tests...
2022-09-27T16:44:54.7091989Z ----------------------------------------------------------------------
2022-09-27T16:44:54.7109838Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-09-27T16:44:54.7110507Z 
2022-09-27T16:44:54.7111107Z ----------------------------------------------------------------------
2022-09-27T16:44:54.7111548Z Ran 1 test in 0.002s
2022-09-27T16:44:54.7111716Z 
2022-09-27T16:44:54.7111825Z OK (skipped=1)
2022-09-27T16:44:54.7111981Z 
2022-09-27T16:44:54.7112088Z Generating XML reports...
2022-09-27T16:44:54.7146255Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164454.xml
2022-09-27T16:44:55.4865564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:55.4866230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:55.4867513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:55.4868284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:55.6838834Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:55.7206715Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116983
2022-09-27T16:44:55.7297691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116984
2022-09-27T16:44:56.0041922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:44:56.0064030Z 
2022-09-27T16:44:56.0064249Z Running tests...
2022-09-27T16:44:56.0064974Z ----------------------------------------------------------------------
2022-09-27T16:44:56.7246167Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:56.7246777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:56.7248977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:56.7249471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:57.2369321Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:57.2392948Z 
2022-09-27T16:44:57.2393188Z Running tests...
2022-09-27T16:44:57.2393625Z ----------------------------------------------------------------------
2022-09-27T16:44:57.2417631Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:44:57.2417986Z 
2022-09-27T16:44:57.2418271Z ----------------------------------------------------------------------
2022-09-27T16:44:57.2418594Z Ran 1 test in 0.002s
2022-09-27T16:44:57.2419093Z 
2022-09-27T16:44:57.2419201Z OK (skipped=1)
2022-09-27T16:44:57.2419360Z 
2022-09-27T16:44:57.2419489Z Generating XML reports...
2022-09-27T16:44:57.2453699Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164457.xml
2022-09-27T16:44:57.3615117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:57.3615616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:57.3617486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:57.3617980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:57.3686429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:57.3686900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:57.3690272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:57.3690765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:57.6078447Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:44:57.6272980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117053
2022-09-27T16:44:57.6278467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117054
2022-09-27T16:44:57.8831955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:57.8880380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:57.8881451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:57.8933007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:57.8939576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:44:57.9896424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:44:59.3845036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:59.3845540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:59.3847745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:59.3848241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:59.3885682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:59.3886162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:59.3889203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:59.3889691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:59.3976977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:44:59.3977427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:44:59.3981035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:44:59.3981514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:44:59.9207330Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:44:59.9231170Z 
2022-09-27T16:44:59.9231382Z Running tests...
2022-09-27T16:44:59.9231820Z ----------------------------------------------------------------------
2022-09-27T16:44:59.9447011Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:44:59.9447659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:44:59.9448394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:59.9449105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:44:59.9587081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:00.0401922Z ok (5.929s)
2022-09-27T16:45:00.0402187Z 
2022-09-27T16:45:00.0402819Z ----------------------------------------------------------------------
2022-09-27T16:45:00.0403176Z Ran 1 test in 5.929s
2022-09-27T16:45:00.0403358Z 
2022-09-27T16:45:00.0403434Z OK
2022-09-27T16:45:00.0403576Z 
2022-09-27T16:45:00.0403710Z Generating XML reports...
2022-09-27T16:45:00.0440305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164454.xml
2022-09-27T16:45:00.0460951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:00.4348056Z skip: Skipped due to small world size. (4.428s)
2022-09-27T16:45:00.4348511Z 
2022-09-27T16:45:00.4348966Z ----------------------------------------------------------------------
2022-09-27T16:45:00.4349292Z Ran 1 test in 4.428s
2022-09-27T16:45:00.4349458Z 
2022-09-27T16:45:00.4349572Z OK (skipped=1)
2022-09-27T16:45:00.4349734Z 
2022-09-27T16:45:00.4349861Z Generating XML reports...
2022-09-27T16:45:00.4384382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164455.xml
2022-09-27T16:45:01.6985013Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:01.7306726Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117240
2022-09-27T16:45:01.7312604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117241
2022-09-27T16:45:02.2061250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:02.2062102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:02.2063378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:02.2063865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:02.6575434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:02.6575998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:02.6577616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:02.6578108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:02.7629836Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:02.7652581Z 
2022-09-27T16:45:02.7652847Z Running tests...
2022-09-27T16:45:02.7653301Z ----------------------------------------------------------------------
2022-09-27T16:45:03.1671676Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:03.1694603Z 
2022-09-27T16:45:03.1695016Z Running tests...
2022-09-27T16:45:03.1695553Z ----------------------------------------------------------------------
2022-09-27T16:45:03.3814338Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:03.3815160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:03.3817178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:03.3817655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:03.3856207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:03.3856678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:03.3860116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:03.3860591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:03.9118125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:03.9142211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:03.9143008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:03.9218868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:03.9226274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:04.0157430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:04.4183519Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:04.4359895Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.670s)
2022-09-27T16:45:04.4360703Z 
2022-09-27T16:45:04.4360989Z ----------------------------------------------------------------------
2022-09-27T16:45:04.4361308Z Ran 1 test in 1.671s
2022-09-27T16:45:04.4361476Z 
2022-09-27T16:45:04.4361587Z OK (skipped=1)
2022-09-27T16:45:04.4361743Z 
2022-09-27T16:45:04.4361870Z Generating XML reports...
2022-09-27T16:45:04.4396680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164502.xml
2022-09-27T16:45:04.8625421Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:04.8835626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117315
2022-09-27T16:45:04.8841022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117316
2022-09-27T16:45:05.4202466Z [1664297105.419688] [28fd6574525b:117241:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:05.4203032Z [1664297105.419688] [28fd6574525b:117241:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:05.4405267Z [1664297105.440010] [28fd6574525b:117240:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:05.4405806Z [1664297105.440010] [28fd6574525b:117240:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:06.3415098Z ok (6.418s)
2022-09-27T16:45:06.3415309Z 
2022-09-27T16:45:06.3415718Z ----------------------------------------------------------------------
2022-09-27T16:45:06.3416056Z Ran 1 test in 6.418s
2022-09-27T16:45:06.3416222Z 
2022-09-27T16:45:06.3416321Z OK
2022-09-27T16:45:06.3416457Z 
2022-09-27T16:45:06.3416597Z Generating XML reports...
2022-09-27T16:45:06.3451926Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164459.xml
2022-09-27T16:45:06.6079735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:06.6081109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:06.6081794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:06.6082475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:06.6321284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:06.6321771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:06.6324430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:06.6325190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:06.6837064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:06.6837559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:06.6840777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:06.6841409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:07.1727010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:07.1727583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:07.1728373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:07.1729097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:07.1735288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:07.1736051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:07.1822951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:07.1847126Z 
2022-09-27T16:45:07.1847474Z Running tests...
2022-09-27T16:45:07.1847910Z ----------------------------------------------------------------------
2022-09-27T16:45:07.1869112Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:45:07.1869409Z 
2022-09-27T16:45:07.1869702Z ----------------------------------------------------------------------
2022-09-27T16:45:07.1870050Z Ran 1 test in 0.002s
2022-09-27T16:45:07.1870198Z 
2022-09-27T16:45:07.1870577Z OK (skipped=1)
2022-09-27T16:45:07.1870753Z 
2022-09-27T16:45:07.1870883Z Generating XML reports...
2022-09-27T16:45:07.1905701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164507.xml
2022-09-27T16:45:07.6909782Z ok (4.521s)
2022-09-27T16:45:07.6910014Z 
2022-09-27T16:45:07.6910418Z ----------------------------------------------------------------------
2022-09-27T16:45:07.6910743Z Ran 1 test in 4.521s
2022-09-27T16:45:07.6910916Z 
2022-09-27T16:45:07.6911019Z OK
2022-09-27T16:45:07.6911160Z 
2022-09-27T16:45:07.6911297Z Generating XML reports...
2022-09-27T16:45:07.6945883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164503.xml
2022-09-27T16:45:08.3643939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:08.3644462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:08.3647421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:08.3647899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:08.8536161Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:08.8559458Z 
2022-09-27T16:45:08.8559879Z Running tests...
2022-09-27T16:45:08.8560549Z ----------------------------------------------------------------------
2022-09-27T16:45:08.8578794Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-09-27T16:45:08.8579417Z 
2022-09-27T16:45:08.8579901Z ----------------------------------------------------------------------
2022-09-27T16:45:08.8580248Z Ran 1 test in 0.002s
2022-09-27T16:45:08.8580397Z 
2022-09-27T16:45:08.8580509Z OK (skipped=1)
2022-09-27T16:45:08.8580671Z 
2022-09-27T16:45:08.8580810Z Generating XML reports...
2022-09-27T16:45:08.8615809Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164508.xml
2022-09-27T16:45:09.3130361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:09.3130852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:09.3132154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:09.3132645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:09.6643734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:09.6644262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:09.6646956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:09.6647472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:09.8460521Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:09.8506310Z 
2022-09-27T16:45:09.8507215Z Running tests...
2022-09-27T16:45:09.8508208Z ----------------------------------------------------------------------
2022-09-27T16:45:09.8558106Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.005s)
2022-09-27T16:45:09.8559666Z 
2022-09-27T16:45:09.8561412Z ----------------------------------------------------------------------
2022-09-27T16:45:09.8562146Z Ran 1 test in 0.005s
2022-09-27T16:45:09.8562542Z 
2022-09-27T16:45:09.8562796Z OK (skipped=1)
2022-09-27T16:45:09.8563146Z 
2022-09-27T16:45:09.8563436Z Generating XML reports...
2022-09-27T16:45:09.8626667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164509.xml
2022-09-27T16:45:10.2127264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:10.2148473Z 
2022-09-27T16:45:10.2148786Z Running tests...
2022-09-27T16:45:10.2149243Z ----------------------------------------------------------------------
2022-09-27T16:45:10.7499139Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:10.7499722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:10.7502848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:11.2494485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:11.2495179Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:11.2517869Z 
2022-09-27T16:45:11.2518151Z Running tests...
2022-09-27T16:45:11.2518589Z ----------------------------------------------------------------------
2022-09-27T16:45:11.2537643Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-09-27T16:45:11.2538592Z 
2022-09-27T16:45:11.2538907Z ----------------------------------------------------------------------
2022-09-27T16:45:11.2539246Z Ran 1 test in 0.002s
2022-09-27T16:45:11.2539413Z 
2022-09-27T16:45:11.2539524Z OK (skipped=1)
2022-09-27T16:45:11.2539664Z 
2022-09-27T16:45:11.2539792Z Generating XML reports...
2022-09-27T16:45:11.2573711Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164511.xml
2022-09-27T16:45:11.7781187Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:11.7825876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:11.7826783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:11.7830948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:11.7831937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:11.7945123Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.579s)
2022-09-27T16:45:11.7946222Z 
2022-09-27T16:45:11.7946788Z ----------------------------------------------------------------------
2022-09-27T16:45:11.7947438Z Ran 1 test in 1.580s
2022-09-27T16:45:11.7947734Z 
2022-09-27T16:45:11.7947945Z OK (skipped=1)
2022-09-27T16:45:11.7948245Z 
2022-09-27T16:45:11.7948509Z Generating XML reports...
2022-09-27T16:45:11.7979614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164510.xml
2022-09-27T16:45:12.3740847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:12.3763935Z 
2022-09-27T16:45:12.3764223Z Running tests...
2022-09-27T16:45:12.3764654Z ----------------------------------------------------------------------
2022-09-27T16:45:12.3785897Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:45:12.3786505Z 
2022-09-27T16:45:12.3786965Z ----------------------------------------------------------------------
2022-09-27T16:45:12.3787345Z Ran 1 test in 0.002s
2022-09-27T16:45:12.3787500Z 
2022-09-27T16:45:12.3787625Z OK (skipped=1)
2022-09-27T16:45:12.3787787Z 
2022-09-27T16:45:12.3787915Z Generating XML reports...
2022-09-27T16:45:12.3823065Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164512.xml
2022-09-27T16:45:13.1832135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:13.1832949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:13.1833887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:13.1834420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:13.7273206Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:13.7295401Z 
2022-09-27T16:45:13.7295591Z Running tests...
2022-09-27T16:45:13.7296045Z ----------------------------------------------------------------------
2022-09-27T16:45:13.7314856Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-09-27T16:45:13.7315407Z 
2022-09-27T16:45:13.7315851Z ----------------------------------------------------------------------
2022-09-27T16:45:13.7316473Z Ran 1 test in 0.002s
2022-09-27T16:45:13.7316639Z 
2022-09-27T16:45:13.7316750Z OK (skipped=1)
2022-09-27T16:45:13.7316909Z 
2022-09-27T16:45:13.7317035Z Generating XML reports...
2022-09-27T16:45:13.7349892Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164513.xml
2022-09-27T16:45:13.9967107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:13.9968092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:13.9969606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:13.9970586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:14.3584327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:14.3585272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:14.3586493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:14.3587041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:14.4971987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:14.4995286Z 
2022-09-27T16:45:14.4995530Z Running tests...
2022-09-27T16:45:14.4995963Z ----------------------------------------------------------------------
2022-09-27T16:45:14.8481174Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:14.8504929Z 
2022-09-27T16:45:14.8505190Z Running tests...
2022-09-27T16:45:14.8505654Z ----------------------------------------------------------------------
2022-09-27T16:45:15.6522342Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:15.6523579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:15.6524736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:15.6525755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:16.1743017Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:16.1765629Z 
2022-09-27T16:45:16.1766084Z Running tests...
2022-09-27T16:45:16.1766586Z ----------------------------------------------------------------------
2022-09-27T16:45:16.1779146Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:16.1976176Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117728
2022-09-27T16:45:16.1982375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117729
2022-09-27T16:45:16.4179137Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:16.4381272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117732
2022-09-27T16:45:16.4386763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117733
2022-09-27T16:45:17.8442616Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:17.8636011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117859
2022-09-27T16:45:17.8642381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117860
2022-09-27T16:45:17.9167406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:17.9167918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:17.9169178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:17.9169966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:17.9347822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:17.9348373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:17.9350601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:17.9351069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:18.1851492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:18.1852032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:18.1853277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:18.1853762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:18.1970676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:18.1971142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:18.1974808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:18.1975278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:18.4610099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:18.4610812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:18.4611598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:18.4612320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:18.4617926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:18.4618432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:18.4726276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:45:18.4727094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:45:18.4727837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:45:18.4728738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:45:18.7733609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:18.7854722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:18.7937544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:18.7938282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:18.7944014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:18.8868350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:19.0051122Z ok (4.505s)
2022-09-27T16:45:19.0051353Z 
2022-09-27T16:45:19.0052044Z ----------------------------------------------------------------------
2022-09-27T16:45:19.0052431Z Ran 1 test in 4.505s
2022-09-27T16:45:19.0052601Z 
2022-09-27T16:45:19.0052987Z OK
2022-09-27T16:45:19.0053129Z 
2022-09-27T16:45:19.0053245Z Generating XML reports...
2022-09-27T16:45:19.0088545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164514.xml
2022-09-27T16:45:19.2455162Z ok (4.395s)
2022-09-27T16:45:19.2455518Z 
2022-09-27T16:45:19.2456108Z ----------------------------------------------------------------------
2022-09-27T16:45:19.2456666Z Ran 1 test in 4.395s
2022-09-27T16:45:19.2456938Z 
2022-09-27T16:45:19.2457091Z OK
2022-09-27T16:45:19.2457318Z 
2022-09-27T16:45:19.2457514Z Generating XML reports...
2022-09-27T16:45:19.2492272Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164514.xml
2022-09-27T16:45:19.5554837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:19.5555352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:19.5557166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:19.5557665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:19.6122291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:19.6123121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:19.6124893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:19.6125662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:20.0584868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:20.1118733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:20.1119540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:20.1192062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:20.1199018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:20.1300107Z STAGE:2022-09-27 16:45:20 117859:117859 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:45:20.2137260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:20.2266225Z STAGE:2022-09-27 16:45:20 117860:117860 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:45:21.1890464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:21.1890986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:21.1893967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:21.1894460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:21.3965357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:21.3965862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:21.3968085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:21.3968559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:21.7128792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:21.7152881Z 
2022-09-27T16:45:21.7153039Z Running tests...
2022-09-27T16:45:21.7153512Z ----------------------------------------------------------------------
2022-09-27T16:45:21.9030622Z   test_get_future (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:21.9053781Z 
2022-09-27T16:45:21.9054171Z Running tests...
2022-09-27T16:45:21.9054662Z ----------------------------------------------------------------------
2022-09-27T16:45:21.9069214Z [1664297121.906639] [28fd6574525b:117859:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:21.9069699Z [1664297121.906639] [28fd6574525b:117859:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:21.9323059Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... [1664297121.931802] [28fd6574525b:117860:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:21.9323690Z [1664297121.931802] [28fd6574525b:117860:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:22.3587935Z STAGE:2022-09-27 16:45:22 117860:117860 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:45:22.3594396Z STAGE:2022-09-27 16:45:22 117859:117859 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:45:23.3757086Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:23.3971323Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118033
2022-09-27T16:45:23.3976817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118034
2022-09-27T16:45:23.4898418Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:23.5096523Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118036
2022-09-27T16:45:23.5101984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118037
2022-09-27T16:45:23.6764931Z ok (7.500s)
2022-09-27T16:45:23.6765130Z 
2022-09-27T16:45:23.6765521Z ----------------------------------------------------------------------
2022-09-27T16:45:23.6765875Z Ran 1 test in 7.500s
2022-09-27T16:45:23.6766048Z 
2022-09-27T16:45:23.6766144Z OK
2022-09-27T16:45:23.6766280Z 
2022-09-27T16:45:23.6766396Z Generating XML reports...
2022-09-27T16:45:23.6801843Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164516.xml
2022-09-27T16:45:25.1987623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:25.1988631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:25.1989827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:25.1991134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:25.2076420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:25.2077404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:25.2080862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:25.2081825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:25.2811437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:25.2812355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:25.2813505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:25.2814430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:25.3449391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:25.3450516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:25.3452122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:25.3453044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:25.7435188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:25.7436265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:25.7437695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:25.7439114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:25.7443748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:25.7444727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:25.7879435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:25.8683197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:25.8684751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:25.8691419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:25.8698336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:25.8972529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:25.8973482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:25.8974763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:25.8975293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:25.9696839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:26.1046560Z ok (4.389s)
2022-09-27T16:45:26.1046771Z 
2022-09-27T16:45:26.1047815Z ----------------------------------------------------------------------
2022-09-27T16:45:26.1048483Z Ran 1 test in 4.389s
2022-09-27T16:45:26.1048796Z 
2022-09-27T16:45:26.1048968Z OK
2022-09-27T16:45:26.1049233Z 
2022-09-27T16:45:26.1049485Z Generating XML reports...
2022-09-27T16:45:26.1084707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164521.xml
2022-09-27T16:45:26.4175137Z ok (4.512s)
2022-09-27T16:45:26.4175488Z 
2022-09-27T16:45:26.4176122Z ----------------------------------------------------------------------
2022-09-27T16:45:26.4176553Z Ran 1 test in 4.512s
2022-09-27T16:45:26.4176765Z 
2022-09-27T16:45:26.4176929Z OK
2022-09-27T16:45:26.4177174Z 
2022-09-27T16:45:26.4177461Z Generating XML reports...
2022-09-27T16:45:26.4197693Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:26.4211599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164521.xml
2022-09-27T16:45:26.4221363Z 
2022-09-27T16:45:26.4221752Z Running tests...
2022-09-27T16:45:26.4222238Z ----------------------------------------------------------------------
2022-09-27T16:45:26.4240381Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:45:26.4240688Z 
2022-09-27T16:45:26.4241232Z ----------------------------------------------------------------------
2022-09-27T16:45:26.4241568Z Ran 1 test in 0.002s
2022-09-27T16:45:26.4241734Z 
2022-09-27T16:45:26.4242081Z OK (skipped=1)
2022-09-27T16:45:26.4242243Z 
2022-09-27T16:45:26.4242362Z Generating XML reports...
2022-09-27T16:45:26.4276550Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164526.xml
2022-09-27T16:45:28.2689382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:28.2689908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:28.2691987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:28.2692486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:28.4961777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:28.4962273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:28.4964378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:28.4964878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:28.5652954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:28.5653436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:28.5656239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:28.5656735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:28.7572169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:28.7594111Z 
2022-09-27T16:45:28.7594310Z Running tests...
2022-09-27T16:45:28.7594747Z ----------------------------------------------------------------------
2022-09-27T16:45:28.9849317Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:28.9871659Z 
2022-09-27T16:45:28.9872061Z Running tests...
2022-09-27T16:45:28.9872767Z ----------------------------------------------------------------------
2022-09-27T16:45:28.9891108Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:45:28.9891454Z 
2022-09-27T16:45:28.9891750Z ----------------------------------------------------------------------
2022-09-27T16:45:28.9892083Z Ran 1 test in 0.002s
2022-09-27T16:45:28.9892253Z 
2022-09-27T16:45:28.9892366Z OK (skipped=1)
2022-09-27T16:45:28.9892523Z 
2022-09-27T16:45:28.9892633Z Generating XML reports...
2022-09-27T16:45:28.9927535Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164528.xml
2022-09-27T16:45:29.0534713Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:29.0557769Z 
2022-09-27T16:45:29.0558069Z Running tests...
2022-09-27T16:45:29.0558546Z ----------------------------------------------------------------------
2022-09-27T16:45:29.0583194Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:45:29.0583580Z 
2022-09-27T16:45:29.0584089Z ----------------------------------------------------------------------
2022-09-27T16:45:29.0584426Z Ran 1 test in 0.003s
2022-09-27T16:45:29.0584587Z 
2022-09-27T16:45:29.0584696Z OK (skipped=1)
2022-09-27T16:45:29.0584852Z 
2022-09-27T16:45:29.0584976Z Generating XML reports...
2022-09-27T16:45:29.0625996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164529.xml
2022-09-27T16:45:30.3721387Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:30.3914555Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118378
2022-09-27T16:45:30.3921303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118379
2022-09-27T16:45:31.0475911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:31.0482187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:31.0484276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:31.0487422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:31.1419109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:31.1420191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:31.1421753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:31.1422741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:31.5975739Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:31.5996911Z 
2022-09-27T16:45:31.5997409Z Running tests...
2022-09-27T16:45:31.5997922Z ----------------------------------------------------------------------
2022-09-27T16:45:31.6017270Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:45:31.6017857Z 
2022-09-27T16:45:31.6018342Z ----------------------------------------------------------------------
2022-09-27T16:45:31.6018684Z Ran 1 test in 0.002s
2022-09-27T16:45:31.6018852Z 
2022-09-27T16:45:31.6018964Z OK (skipped=1)
2022-09-27T16:45:31.6019130Z 
2022-09-27T16:45:31.6019259Z Generating XML reports...
2022-09-27T16:45:31.6052439Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164531.xml
2022-09-27T16:45:31.6272670Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:31.6295636Z 
2022-09-27T16:45:31.6295913Z Running tests...
2022-09-27T16:45:31.6296345Z ----------------------------------------------------------------------
2022-09-27T16:45:31.6316266Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:45:31.6317187Z 
2022-09-27T16:45:31.6318071Z ----------------------------------------------------------------------
2022-09-27T16:45:31.6318715Z Ran 1 test in 0.002s
2022-09-27T16:45:31.6319039Z 
2022-09-27T16:45:31.6319268Z OK (skipped=1)
2022-09-27T16:45:31.6319560Z 
2022-09-27T16:45:31.6319803Z Generating XML reports...
2022-09-27T16:45:31.6353890Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164531.xml
2022-09-27T16:45:32.0755845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:32.0756864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:32.0758045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:32.0758977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:32.0778734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:32.0779624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:32.0782184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:32.0783130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:32.6778217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:32.6779626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:32.6781121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:32.6782513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:32.6784983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:32.6785951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:33.0986091Z ok (4.339s)
2022-09-27T16:45:33.0986345Z 
2022-09-27T16:45:33.0986764Z ----------------------------------------------------------------------
2022-09-27T16:45:33.0987112Z Ran 1 test in 4.339s
2022-09-27T16:45:33.0987284Z 
2022-09-27T16:45:33.0987384Z OK
2022-09-27T16:45:33.0987525Z 
2022-09-27T16:45:33.0987645Z Generating XML reports...
2022-09-27T16:45:33.1022897Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164528.xml
2022-09-27T16:45:33.5271341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:33.5271901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:33.5274352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:33.5274850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:33.5758276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:33.5758743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:33.5762381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:33.5762885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:34.0189809Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:34.0212217Z 
2022-09-27T16:45:34.0212763Z Running tests...
2022-09-27T16:45:34.0213394Z ----------------------------------------------------------------------
2022-09-27T16:45:34.0557763Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:34.0580903Z 
2022-09-27T16:45:34.0581228Z Running tests...
2022-09-27T16:45:34.0582035Z ----------------------------------------------------------------------
2022-09-27T16:45:35.1997676Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:35.1998873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:35.2000091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:35.2001443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:35.6362041Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:35.6421655Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:35.6527864Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.631s)
2022-09-27T16:45:35.6528982Z 
2022-09-27T16:45:35.6529555Z ----------------------------------------------------------------------
2022-09-27T16:45:35.6530180Z Ran 1 test in 1.631s
2022-09-27T16:45:35.6530909Z 
2022-09-27T16:45:35.6531123Z OK (skipped=1)
2022-09-27T16:45:35.6531421Z 
2022-09-27T16:45:35.6531666Z Generating XML reports...
2022-09-27T16:45:35.6561822Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164534.xml
2022-09-27T16:45:35.6761712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118556
2022-09-27T16:45:35.6767125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118557
2022-09-27T16:45:35.7333609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:35.7357550Z 
2022-09-27T16:45:35.7357808Z Running tests...
2022-09-27T16:45:35.7358239Z ----------------------------------------------------------------------
2022-09-27T16:45:37.3540292Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:37.3541492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:37.3542703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:37.3543615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:37.3674789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:37.3675714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:37.3679268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:37.3680556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:37.3961983Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:37.4163929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118656
2022-09-27T16:45:37.4170441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118657
2022-09-27T16:45:37.8293125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:37.8294181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:37.8295395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:37.8296368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:37.8662414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:37.8828798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:37.8830288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:37.8864804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:37.8871627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:37.8939101Z STAGE:2022-09-27 16:45:37 118557:118557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:45:37.9843805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:37.9910155Z STAGE:2022-09-27 16:45:37 118556:118556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:45:38.3813189Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:38.3836871Z 
2022-09-27T16:45:38.3837315Z Running tests...
2022-09-27T16:45:38.3837817Z ----------------------------------------------------------------------
2022-09-27T16:45:39.0563246Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:39.0564218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:39.0565221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:39.0565696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:39.1130173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:39.1130682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:39.1131893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:39.1132364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:39.6835263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:39.6835881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:39.6836680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:39.6837390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:39.6844327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:39.6844832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:39.7056725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:45:39.7057271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:45:39.7058015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:45:39.7058711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:45:39.7512423Z [1664297139.750718] [28fd6574525b:118556:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:39.7512963Z [1664297139.750718] [28fd6574525b:118556:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:39.7616567Z [1664297139.761188] [28fd6574525b:118557:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:39.7617328Z [1664297139.761188] [28fd6574525b:118557:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:40.0058768Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:40.0261790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118754
2022-09-27T16:45:40.0266742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118755
2022-09-27T16:45:40.1235822Z ok (4.387s)
2022-09-27T16:45:40.1236036Z 
2022-09-27T16:45:40.1236423Z ----------------------------------------------------------------------
2022-09-27T16:45:40.1236754Z Ran 1 test in 4.388s
2022-09-27T16:45:40.1236923Z 
2022-09-27T16:45:40.1237031Z OK
2022-09-27T16:45:40.1237168Z 
2022-09-27T16:45:40.1237307Z Generating XML reports...
2022-09-27T16:45:40.1273615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164535.xml
2022-09-27T16:45:40.1570810Z STAGE:2022-09-27 16:45:40 118557:118557 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:45:40.1589292Z STAGE:2022-09-27 16:45:40 118556:118556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:45:41.4904389Z ok (7.432s)
2022-09-27T16:45:41.4904636Z 
2022-09-27T16:45:41.4905029Z ----------------------------------------------------------------------
2022-09-27T16:45:41.4905375Z Ran 1 test in 7.432s
2022-09-27T16:45:41.4905539Z 
2022-09-27T16:45:41.4905638Z OK
2022-09-27T16:45:41.4905758Z 
2022-09-27T16:45:41.4905897Z Generating XML reports...
2022-09-27T16:45:41.4960931Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164534.xml
2022-09-27T16:45:41.7287163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:41.7287894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:41.7289138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:41.7289664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:41.7909626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:41.7910188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:41.7911300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:41.7911789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:42.2324045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:42.2351961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:42.2352443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:42.2355294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:42.2355798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:42.3136952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:42.3137785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:42.3236338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:42.3242817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:42.4150475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:42.7842658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:42.7865382Z 
2022-09-27T16:45:42.7865594Z Running tests...
2022-09-27T16:45:42.7866487Z ----------------------------------------------------------------------
2022-09-27T16:45:43.5817290Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:43.5817889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:43.5819180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:43.5820013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:43.7183311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p1bwt1i
2022-09-27T16:45:43.7184494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p1bwt1i/_remote_module_non_scriptable.py
2022-09-27T16:45:43.8128343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpldhr0hl2
2022-09-27T16:45:43.8129590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpldhr0hl2/_remote_module_non_scriptable.py
2022-09-27T16:45:44.0940537Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:44.0962580Z 
2022-09-27T16:45:44.0962908Z Running tests...
2022-09-27T16:45:44.0963850Z ----------------------------------------------------------------------
2022-09-27T16:45:44.4445364Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:44.4648801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118900
2022-09-27T16:45:44.4653358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118901
2022-09-27T16:45:45.7597432Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:45.7798592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118974
2022-09-27T16:45:45.7803876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118975
2022-09-27T16:45:46.1405722Z ok (7.756s)
2022-09-27T16:45:46.1405950Z 
2022-09-27T16:45:46.1406366Z ----------------------------------------------------------------------
2022-09-27T16:45:46.1406707Z Ran 1 test in 7.757s
2022-09-27T16:45:46.1406882Z 
2022-09-27T16:45:46.1406959Z OK
2022-09-27T16:45:46.1407094Z 
2022-09-27T16:45:46.1407228Z Generating XML reports...
2022-09-27T16:45:46.1459261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164538.xml
2022-09-27T16:45:46.1607893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:46.1608364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:46.1611104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:46.1611599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:46.1624898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:46.1625359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:46.1628908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:46.7499579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:46.7500345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:46.7500871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:46.7501671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:46.7502589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:46.7608258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:46.8512281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:46.8719591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:45:46.8720559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:45:46.8721352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:45:46.8722066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:45:47.2739322Z ok (4.487s)
2022-09-27T16:45:47.2740500Z 
2022-09-27T16:45:47.2741793Z ----------------------------------------------------------------------
2022-09-27T16:45:47.2742895Z Ran 1 test in 4.487s
2022-09-27T16:45:47.2743583Z 
2022-09-27T16:45:47.2744564Z OK
2022-09-27T16:45:47.2745275Z 
2022-09-27T16:45:47.2745818Z Generating XML reports...
2022-09-27T16:45:47.2802537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164542.xml
2022-09-27T16:45:47.4931482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:47.4931994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:47.4934264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:47.4934743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:47.5220966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:47.5221456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:47.5223627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:47.5224110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:48.0099138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:48.0669576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:48.0670409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:48.0706682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:48.0713502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:48.1680909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:48.2828143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:48.2828666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:48.2829751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:48.2830222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:48.8648403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:48.8670322Z 
2022-09-27T16:45:48.8670469Z Running tests...
2022-09-27T16:45:48.8671663Z ----------------------------------------------------------------------
2022-09-27T16:45:49.4813876Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:49.4814503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:49.4815113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:49.4815604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:49.5438026Z [1664297149.543313] [28fd6574525b:118975:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:49.5438548Z [1664297149.543313] [28fd6574525b:118975:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:49.5647678Z [1664297149.564291] [28fd6574525b:118974:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:49.5648255Z [1664297149.564291] [28fd6574525b:118974:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:50.0001524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:50.0025456Z 
2022-09-27T16:45:50.0025739Z Running tests...
2022-09-27T16:45:50.0026223Z ----------------------------------------------------------------------
2022-09-27T16:45:50.4917780Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... ok (6.395s)
2022-09-27T16:45:50.4918090Z 
2022-09-27T16:45:50.4918480Z ----------------------------------------------------------------------
2022-09-27T16:45:50.4918822Z Ran 1 test in 6.395s
2022-09-27T16:45:50.4918970Z 
2022-09-27T16:45:50.4919063Z OK
2022-09-27T16:45:50.4919200Z 
2022-09-27T16:45:50.4919331Z Generating XML reports...
2022-09-27T16:45:50.4954478Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164544.xml
2022-09-27T16:45:50.5109220Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:50.5316908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119134
2022-09-27T16:45:50.5322054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119135
2022-09-27T16:45:51.8576637Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:51.8773150Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119231
2022-09-27T16:45:51.8779354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119232
2022-09-27T16:45:52.1736934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:52.1737946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:52.1739084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:52.1740057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:52.2849357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:52.2850366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:52.2854447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:52.2855414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:52.7363339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:52.7363845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:52.7365138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:52.7365633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:52.7700574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:52.9001295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:52.9002168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:52.9017909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:52.9025745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:53.0014993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:53.2320516Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:45:53.2343776Z 
2022-09-27T16:45:53.2344208Z Running tests...
2022-09-27T16:45:53.2344734Z ----------------------------------------------------------------------
2022-09-27T16:45:53.5755003Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:53.5756155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:53.5757714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:53.5758203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:53.6312392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:53.6312909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:53.6314685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:53.6315191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:54.1771241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:54.1771811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:54.1772623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:54.1773331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:54.1779820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:54.1780326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:54.9323787Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:45:54.9535045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119323
2022-09-27T16:45:54.9540812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119324
2022-09-27T16:45:55.5581727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2rutnc5
2022-09-27T16:45:55.5582372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2rutnc5/_remote_module_non_scriptable.py
2022-09-27T16:45:55.5812546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwagqdjc8
2022-09-27T16:45:55.5814033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwagqdjc8/_remote_module_non_scriptable.py
2022-09-27T16:45:56.1466296Z ok (7.279s)
2022-09-27T16:45:56.1466495Z 
2022-09-27T16:45:56.1466902Z ----------------------------------------------------------------------
2022-09-27T16:45:56.1467251Z Ran 1 test in 7.279s
2022-09-27T16:45:56.1467420Z 
2022-09-27T16:45:56.1467518Z OK
2022-09-27T16:45:56.1467659Z 
2022-09-27T16:45:56.1467778Z Generating XML reports...
2022-09-27T16:45:56.1521762Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164548.xml
2022-09-27T16:45:56.6062146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:56.6062673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:56.6064707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:56.6065205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:56.6905645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:56.6906142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:56.6907177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:56.6907666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:56.7889845Z ok (6.786s)
2022-09-27T16:45:56.7890050Z 
2022-09-27T16:45:56.7890752Z ----------------------------------------------------------------------
2022-09-27T16:45:56.7891106Z Ran 1 test in 6.786s
2022-09-27T16:45:56.7891316Z 
2022-09-27T16:45:56.7891415Z OK
2022-09-27T16:45:56.7891552Z 
2022-09-27T16:45:56.7891668Z Generating XML reports...
2022-09-27T16:45:56.7926172Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164549.xml
2022-09-27T16:45:57.1311678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:45:57.2182111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:45:57.2183544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:57.2224121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:45:57.2231100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:45:57.2296397Z STAGE:2022-09-27 16:45:57 119323:119323 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:45:57.3239509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:45:57.3364305Z STAGE:2022-09-27 16:45:57 119324:119324 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:45:58.4144730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:58.4145227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:58.4146317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:58.4146809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:58.9413622Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:45:58.9436703Z 
2022-09-27T16:45:58.9437129Z Running tests...
2022-09-27T16:45:58.9437581Z ----------------------------------------------------------------------
2022-09-27T16:45:58.9539913Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:45:58.9540487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:45:58.9543544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:45:58.9544040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:45:59.0736442Z [1664297159.073137] [28fd6574525b:119323:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:59.0736985Z [1664297159.073137] [28fd6574525b:119323:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:59.1284571Z [1664297159.127973] [28fd6574525b:119324:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:45:59.1285078Z [1664297159.127973] [28fd6574525b:119324:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:45:59.4560602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:45:59.4583234Z 
2022-09-27T16:45:59.4583671Z Running tests...
2022-09-27T16:45:59.4584154Z ----------------------------------------------------------------------
2022-09-27T16:45:59.5065829Z   test_irecv (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:45:59 119323:119323 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:45:59.5066544Z STAGE:2022-09-27 16:45:59 119324:119324 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:46:00.5841198Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:00.6050257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119482
2022-09-27T16:46:00.6055462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119483
2022-09-27T16:46:00.8668302Z ok (7.632s)
2022-09-27T16:46:00.8668480Z 
2022-09-27T16:46:00.8668870Z ----------------------------------------------------------------------
2022-09-27T16:46:00.8669194Z Ran 1 test in 7.632s
2022-09-27T16:46:00.8669365Z 
2022-09-27T16:46:00.8669464Z OK
2022-09-27T16:46:00.8669605Z 
2022-09-27T16:46:00.8669735Z Generating XML reports...
2022-09-27T16:46:00.8705948Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164553.xml
2022-09-27T16:46:01.1123058Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:01.1323692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119485
2022-09-27T16:46:01.1329925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119486
2022-09-27T16:46:02.3971119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:02.3971678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:02.3972992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:02.3973611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:02.4366042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:02.4366575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:02.4368186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:02.4368860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:02.8979392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:02.8979905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:02.8982091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:02.8982865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:02.9059136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:02.9059751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:02.9063560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:02.9064126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:02.9110602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:02.9618271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:02.9619070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:02.9619783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:02.9626327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:02.9627236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:03.0042729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:03.0043248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:03.0045793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:03.0046574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:03.4525527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:03.4526088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:03.4526894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:03.4527591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:03.4534288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:03.4535194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:03.5410756Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:03.5434680Z 
2022-09-27T16:46:03.5434817Z Running tests...
2022-09-27T16:46:03.5436215Z ----------------------------------------------------------------------
2022-09-27T16:46:03.9401144Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... ok (4.481s)
2022-09-27T16:46:03.9401799Z 
2022-09-27T16:46:03.9402520Z ----------------------------------------------------------------------
2022-09-27T16:46:03.9403093Z Ran 1 test in 4.482s
2022-09-27T16:46:03.9403392Z 
2022-09-27T16:46:03.9403529Z OK
2022-09-27T16:46:03.9403792Z 
2022-09-27T16:46:03.9404007Z Generating XML reports...
2022-09-27T16:46:03.9437962Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164559.xml
2022-09-27T16:46:05.1959936Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:05.2169341Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119701
2022-09-27T16:46:05.2258917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119702
2022-09-27T16:46:06.2178160Z ok (7.274s)
2022-09-27T16:46:06.2178517Z 
2022-09-27T16:46:06.2179188Z ----------------------------------------------------------------------
2022-09-27T16:46:06.2179825Z Ran 1 test in 7.274s
2022-09-27T16:46:06.2180143Z 
2022-09-27T16:46:06.2180315Z OK
2022-09-27T16:46:06.2180571Z 
2022-09-27T16:46:06.2180802Z Generating XML reports...
2022-09-27T16:46:06.2214163Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164558.xml
2022-09-27T16:46:06.2669897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:06.2670921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:06.2672100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:06.2673084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:06.7622870Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:46:06.7644752Z 
2022-09-27T16:46:06.7645178Z Running tests...
2022-09-27T16:46:06.7646054Z ----------------------------------------------------------------------
2022-09-27T16:46:06.9270715Z   test_isend (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:06.9271839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:06.9273053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:06.9273580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:06.9851730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:06.9852248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:06.9854268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:06.9854768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:07.5010605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:07.6234196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:07.6235063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:07.6327562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:07.6334204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:07.6401968Z STAGE:2022-09-27 16:46:07 119701:119701 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:46:07.7247731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:07.7315884Z STAGE:2022-09-27 16:46:07 119702:119702 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:46:08.3369855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:08.3370818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:08.3372024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:08.3373002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:08.3897260Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:08.4101994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119818
2022-09-27T16:46:08.4107728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119819
2022-09-27T16:46:08.8621435Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:46:08.8645394Z 
2022-09-27T16:46:08.8645728Z Running tests...
2022-09-27T16:46:08.8646165Z ----------------------------------------------------------------------
2022-09-27T16:46:09.4505652Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... [1664297169.449948] [28fd6574525b:119701:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:46:09.4507206Z [1664297169.449948] [28fd6574525b:119701:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:46:09.4829879Z [1664297169.482400] [28fd6574525b:119702:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:46:09.4831053Z [1664297169.482400] [28fd6574525b:119702:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:46:09.8745906Z STAGE:2022-09-27 16:46:09 119701:119701 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:46:09.8747139Z STAGE:2022-09-27 16:46:09 119702:119702 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:46:10.0904364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:10.0905631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:10.0907166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:10.0908359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:10.1365667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:10.1366528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:10.1367580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:10.1368418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:10.4906929Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:10.5100541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119891
2022-09-27T16:46:10.5106701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119892
2022-09-27T16:46:10.6859944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:10.6860516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:10.6861336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:10.6862053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:10.6867929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:10.6868429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:11.1173982Z ok (4.353s)
2022-09-27T16:46:11.1174210Z 
2022-09-27T16:46:11.1174816Z ----------------------------------------------------------------------
2022-09-27T16:46:11.1175200Z Ran 1 test in 4.353s
2022-09-27T16:46:11.1175372Z 
2022-09-27T16:46:11.1175466Z OK
2022-09-27T16:46:11.1175606Z 
2022-09-27T16:46:11.1175753Z Generating XML reports...
2022-09-27T16:46:11.1210934Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164606.xml
2022-09-27T16:46:11.2387047Z ok (7.695s)
2022-09-27T16:46:11.2388472Z 
2022-09-27T16:46:11.2389654Z ----------------------------------------------------------------------
2022-09-27T16:46:11.2390922Z Ran 1 test in 7.695s
2022-09-27T16:46:11.2391867Z 
2022-09-27T16:46:11.2392311Z OK
2022-09-27T16:46:11.2392865Z 
2022-09-27T16:46:11.2393509Z Generating XML reports...
2022-09-27T16:46:11.2443940Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164603.xml
2022-09-27T16:46:12.1327459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:12.1328468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:12.1329651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:12.1330168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:12.1598963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:12.1599457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:12.1602853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:12.1603348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:12.6377431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:12.6736540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:12.6737459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:12.6782703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:12.6789596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:12.7750460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:13.3378128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:13.3378763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:13.3379953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:13.3380452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:13.4042127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:13.4042642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:13.4045904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:13.4046385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:13.8517552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:46:13.8541613Z 
2022-09-27T16:46:13.8541904Z Running tests...
2022-09-27T16:46:13.8542334Z ----------------------------------------------------------------------
2022-09-27T16:46:13.9232920Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:13.9258481Z 
2022-09-27T16:46:13.9259026Z Running tests...
2022-09-27T16:46:13.9259590Z ----------------------------------------------------------------------
2022-09-27T16:46:13.9277960Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-09-27T16:46:13.9278313Z 
2022-09-27T16:46:13.9278620Z ----------------------------------------------------------------------
2022-09-27T16:46:13.9278960Z Ran 1 test in 0.002s
2022-09-27T16:46:13.9279131Z 
2022-09-27T16:46:13.9279246Z OK (skipped=1)
2022-09-27T16:46:13.9279410Z 
2022-09-27T16:46:13.9279524Z Generating XML reports...
2022-09-27T16:46:13.9315946Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164613.xml
2022-09-27T16:46:14.0197396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2p_o1otn
2022-09-27T16:46:14.0198867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2p_o1otn/_remote_module_non_scriptable.py
2022-09-27T16:46:14.0601666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr2buzon
2022-09-27T16:46:14.0602631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr2buzon/_remote_module_non_scriptable.py
2022-09-27T16:46:14.5804629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:14.5805242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:14.5822986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:14.5823522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:15.2221088Z ok (6.357s)
2022-09-27T16:46:15.2221302Z 
2022-09-27T16:46:15.2221721Z ----------------------------------------------------------------------
2022-09-27T16:46:15.2222074Z Ran 1 test in 6.358s
2022-09-27T16:46:15.2222227Z 
2022-09-27T16:46:15.2222324Z OK
2022-09-27T16:46:15.2222483Z 
2022-09-27T16:46:15.2222623Z Generating XML reports...
2022-09-27T16:46:15.2258442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164608.xml
2022-09-27T16:46:15.5501482Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:15.5700482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120084
2022-09-27T16:46:15.5706609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120085
2022-09-27T16:46:15.9102065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:15.9104830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:15.9106034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:15.9106972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:16.5470647Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:16.5494594Z 
2022-09-27T16:46:16.5495165Z Running tests...
2022-09-27T16:46:16.5496044Z ----------------------------------------------------------------------
2022-09-27T16:46:16.5515426Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-09-27T16:46:16.5516107Z 
2022-09-27T16:46:16.5516664Z ----------------------------------------------------------------------
2022-09-27T16:46:16.5517288Z Ran 1 test in 0.002s
2022-09-27T16:46:16.5517611Z 
2022-09-27T16:46:16.5517814Z OK (skipped=1)
2022-09-27T16:46:16.5518087Z 
2022-09-27T16:46:16.5518325Z Generating XML reports...
2022-09-27T16:46:16.5554130Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164616.xml
2022-09-27T16:46:17.2770588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:17.2771087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:17.2772283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:17.2772789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:17.2804449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:17.2804903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:17.2808525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:17.2809010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:17.3641780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:17.3642583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:17.3644354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:17.3644857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:17.8784183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:17.8784735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:17.8785524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:17.8786237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:17.8892413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:17.8957291Z STAGE:2022-09-27 16:46:17 120085:120085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:46:17.9182603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:46:17.9206579Z 
2022-09-27T16:46:17.9206845Z Running tests...
2022-09-27T16:46:17.9207271Z ----------------------------------------------------------------------
2022-09-27T16:46:17.9797480Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:18.0054792Z STAGE:2022-09-27 16:46:18 120084:120084 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:46:18.0088461Z STAGE:2022-09-27 16:46:18 120084:120084 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:46:18.0091514Z STAGE:2022-09-27 16:46:18 120085:120085 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:46:18.4790272Z ok (4.625s)
2022-09-27T16:46:18.4790483Z 
2022-09-27T16:46:18.4790881Z ----------------------------------------------------------------------
2022-09-27T16:46:18.4791246Z Ran 1 test in 4.625s
2022-09-27T16:46:18.4791412Z 
2022-09-27T16:46:18.4791492Z OK
2022-09-27T16:46:18.4791635Z 
2022-09-27T16:46:18.4791774Z Generating XML reports...
2022-09-27T16:46:18.4849859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164613.xml
2022-09-27T16:46:18.5388216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:18.5388712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:18.5392924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:18.5393422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:19.0659720Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:19.0683071Z 
2022-09-27T16:46:19.0683276Z Running tests...
2022-09-27T16:46:19.0683760Z ----------------------------------------------------------------------
2022-09-27T16:46:19.5688183Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:19.5888894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120233
2022-09-27T16:46:19.5894814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120234
2022-09-27T16:46:20.7863157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:20.7863708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:20.7865729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:20.7866581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:20.8312855Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:20.8512415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120330
2022-09-27T16:46:20.8519405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120331
2022-09-27T16:46:21.2875437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:21.2875958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:21.2878107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:21.2878617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:21.3064415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:46:21.3089500Z 
2022-09-27T16:46:21.3089945Z Running tests...
2022-09-27T16:46:21.3090457Z ----------------------------------------------------------------------
2022-09-27T16:46:21.3393448Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:21.3394422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:21.3396673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:21.3397177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:21.8676491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:21.9281661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:21.9282490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:21.9284716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:21.9292688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:22.0296149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:22.4825993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:22.4826511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:22.4827375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:22.4827868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:22.5315700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:22.5316179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:22.5317474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:22.5317959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:23.0134769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:23.0376058Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:23.0585124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120408
2022-09-27T16:46:23.0590576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120409
2022-09-27T16:46:23.0755948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:23.0757043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:23.0846726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:23.0853130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:23.1770786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:23.3158765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5d1_tdf
2022-09-27T16:46:23.3159391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5d1_tdf/_remote_module_non_scriptable.py
2022-09-27T16:46:23.4021508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabv24l_h
2022-09-27T16:46:23.4022499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabv24l_h/_remote_module_non_scriptable.py
2022-09-27T16:46:24.0066159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:24.0066722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:24.0083384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:24.0087504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:24.6015332Z [1664297184.601070] [28fd6574525b:120331:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:46:24.6015904Z [1664297184.601070] [28fd6574525b:120331:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:46:24.6016397Z [1664297184.601307] [28fd6574525b:120330:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:46:24.6016902Z [1664297184.601307] [28fd6574525b:120330:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:46:24.7006493Z ok (6.780s)
2022-09-27T16:46:24.7006747Z 
2022-09-27T16:46:24.7007144Z ----------------------------------------------------------------------
2022-09-27T16:46:24.7007511Z Ran 1 test in 6.780s
2022-09-27T16:46:24.7007681Z 
2022-09-27T16:46:24.7007786Z OK
2022-09-27T16:46:24.7007927Z 
2022-09-27T16:46:24.7008046Z Generating XML reports...
2022-09-27T16:46:24.7043542Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164617.xml
2022-09-27T16:46:24.7355872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:24.7356373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:24.7358513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:24.7359028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:24.8103853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:24.8104372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:24.8106475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:24.8106976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:25.3637024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:25.3637610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:25.3638436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:25.3639398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:25.3745791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:25.3814680Z STAGE:2022-09-27 16:46:25 120409:120409 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:46:25.4651519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:25.4717589Z STAGE:2022-09-27 16:46:25 120408:120408 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:46:25.4755561Z STAGE:2022-09-27 16:46:25 120409:120409 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:46:25.5750092Z STAGE:2022-09-27 16:46:25 120408:120408 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:46:26.0661379Z ok (4.757s)
2022-09-27T16:46:26.0661808Z 
2022-09-27T16:46:26.0662574Z ----------------------------------------------------------------------
2022-09-27T16:46:26.0663090Z Ran 1 test in 4.757s
2022-09-27T16:46:26.0663243Z 
2022-09-27T16:46:26.0663342Z OK
2022-09-27T16:46:26.0663481Z 
2022-09-27T16:46:26.0663619Z Generating XML reports...
2022-09-27T16:46:26.0699191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164621.xml
2022-09-27T16:46:26.1636884Z ok (7.095s)
2022-09-27T16:46:26.1637251Z 
2022-09-27T16:46:26.1637957Z ----------------------------------------------------------------------
2022-09-27T16:46:26.1638614Z Ran 1 test in 7.095s
2022-09-27T16:46:26.1638792Z 
2022-09-27T16:46:26.1638890Z OK
2022-09-27T16:46:26.1639033Z 
2022-09-27T16:46:26.1639152Z Generating XML reports...
2022-09-27T16:46:26.1675046Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164619.xml
2022-09-27T16:46:26.8526984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:26.8527954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:26.8529944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:26.8530921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:27.4083069Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:46:27.4106977Z 
2022-09-27T16:46:27.4107619Z Running tests...
2022-09-27T16:46:27.4108129Z ----------------------------------------------------------------------
2022-09-27T16:46:28.2095894Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:28.2096563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:28.2097997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:28.2098502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:28.2572090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:28.2572579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:28.2575008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:28.2575516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:28.7128674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:46:28.7154740Z 
2022-09-27T16:46:28.7154993Z Running tests...
2022-09-27T16:46:28.7155457Z ----------------------------------------------------------------------
2022-09-27T16:46:28.7556703Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:28.7580068Z 
2022-09-27T16:46:28.7580409Z Running tests...
2022-09-27T16:46:28.7581175Z ----------------------------------------------------------------------
2022-09-27T16:46:28.9921536Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:29.0127273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120617
2022-09-27T16:46:29.0132033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120618
2022-09-27T16:46:30.3803496Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:30.4005215Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120682
2022-09-27T16:46:30.4010544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120683
2022-09-27T16:46:30.4150963Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:30.4352685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120685
2022-09-27T16:46:30.4358075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120686
2022-09-27T16:46:30.6507050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:30.6507585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:30.6509380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:30.6509888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:30.6634175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:30.6634666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:30.6638321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:30.6638825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:31.2485396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:31.2699769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:31.2701354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:31.2790031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:31.2796291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:31.3713861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:31.7202071Z skip: Need at least 4 CUDA devices (4.309s)
2022-09-27T16:46:31.7202548Z 
2022-09-27T16:46:31.7203192Z ----------------------------------------------------------------------
2022-09-27T16:46:31.7203844Z Ran 1 test in 4.309s
2022-09-27T16:46:31.7204125Z 
2022-09-27T16:46:31.7204333Z OK (skipped=1)
2022-09-27T16:46:31.7204616Z 
2022-09-27T16:46:31.7204840Z Generating XML reports...
2022-09-27T16:46:31.7238419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164627.xml
2022-09-27T16:46:32.0511737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:32.0512707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:32.0513912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:32.0515248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:32.1015126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:32.1016934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:32.1018159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:32.1019131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:32.1331591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:32.1332454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:32.1333873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:32.1334684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:32.1685005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:32.1686013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:32.1687655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:32.1688613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:32.6145087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:32.6422783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:32.6423595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:32.6448752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:32.6455841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:32.6558017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82q0xyp1
2022-09-27T16:46:32.6561316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82q0xyp1/_remote_module_non_scriptable.py
2022-09-27T16:46:32.7020417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:32.7020930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:32.7021673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:32.7022379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:32.7130367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:32.7133660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:46:32.7434438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:32.7534577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmem178p_
2022-09-27T16:46:32.7537377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmem178p_/_remote_module_non_scriptable.py
2022-09-27T16:46:32.8061658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:32.8071611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:46:32.8073464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:46:32.8155314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:46:32.8222718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:46:32.8225977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:46:32.8230410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:46:32.8232129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:46:33.8603906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:33.8604442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:33.8606227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:33.8606983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:34.2595441Z [1664297194.258990] [28fd6574525b:120685:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:46:34.2596537Z [1664297194.258990] [28fd6574525b:120685:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:46:34.3008028Z [1664297194.300321] [28fd6574525b:120686:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:46:34.3008946Z [1664297194.300321] [28fd6574525b:120686:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:46:34.4109741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:46:34.4133133Z 
2022-09-27T16:46:34.4133884Z Running tests...
2022-09-27T16:46:34.4134843Z ----------------------------------------------------------------------
2022-09-27T16:46:35.2464278Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... ok (6.488s)
2022-09-27T16:46:35.2464684Z 
2022-09-27T16:46:35.2465070Z ----------------------------------------------------------------------
2022-09-27T16:46:35.2465414Z Ran 1 test in 6.488s
2022-09-27T16:46:35.2465584Z 
2022-09-27T16:46:35.2465662Z OK
2022-09-27T16:46:35.2465802Z 
2022-09-27T16:46:35.2465928Z Generating XML reports...
2022-09-27T16:46:35.2502148Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164628.xml
2022-09-27T16:46:35.6119510Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:46:36.1259666Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:36.1457643Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120898
2022-09-27T16:46:36.1463440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120899
2022-09-27T16:46:37.4380851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:37.4381392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:37.4383611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:37.4384091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:37.7660994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:37.7661505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:37.7663379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:37.7663853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:37.8049606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:37.8050111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:37.8053061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:37.8053552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:37.9261660Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:37.9283709Z 
2022-09-27T16:46:37.9283827Z Running tests...
2022-09-27T16:46:37.9284719Z ----------------------------------------------------------------------
2022-09-27T16:46:37.9306928Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl & Gloo backend support DistributedDataParallel (0.002s)
2022-09-27T16:46:37.9307513Z 
2022-09-27T16:46:37.9307925Z ----------------------------------------------------------------------
2022-09-27T16:46:37.9308271Z Ran 1 test in 0.002s
2022-09-27T16:46:37.9308439Z 
2022-09-27T16:46:37.9308552Z OK (skipped=1)
2022-09-27T16:46:37.9308694Z 
2022-09-27T16:46:37.9309120Z Generating XML reports...
2022-09-27T16:46:37.9341855Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164637.xml
2022-09-27T16:46:38.2680850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:38.3153372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:38.3154213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:38.3186883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:38.3193865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:38.4167927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:38.7525511Z skip: Need at least 4 CUDA devices (4.339s)
2022-09-27T16:46:38.7525787Z 
2022-09-27T16:46:38.7526187Z ----------------------------------------------------------------------
2022-09-27T16:46:38.7526514Z Ran 1 test in 4.339s
2022-09-27T16:46:38.7526684Z 
2022-09-27T16:46:38.7526795Z OK (skipped=1)
2022-09-27T16:46:38.7526952Z 
2022-09-27T16:46:38.7527083Z Generating XML reports...
2022-09-27T16:46:38.7564477Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164634.xml
2022-09-27T16:46:39.8458487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:39.8469614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:39.8478436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:39.8487369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:40.3762657Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:40.3785950Z 
2022-09-27T16:46:40.3786359Z Running tests...
2022-09-27T16:46:40.3786850Z ----------------------------------------------------------------------
2022-09-27T16:46:40.8186974Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:40.8187604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:40.8189718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:40.8190537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:41.3111661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:46:41.3134587Z 
2022-09-27T16:46:41.3134997Z Running tests...
2022-09-27T16:46:41.3135489Z ----------------------------------------------------------------------
2022-09-27T16:46:41.9834517Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:42.0035713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121071
2022-09-27T16:46:42.0042088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121072
2022-09-27T16:46:42.9262503Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:42.9458061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121136
2022-09-27T16:46:42.9463528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121137
2022-09-27T16:46:43.6471130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:43.6472154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:43.6476724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:43.6477732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:43.7283594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:43.7284099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:43.7285533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:43.7286023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:44.1650286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:44.2348393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:44.2349219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:44.2360332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:44.2367354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:44.3361948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:44.5474070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:44.5474564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:44.5477345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:44.5477850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:44.5642988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:44.5643449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:44.5646819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:44.5647312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:45.0790430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:45.0792385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:45.0793420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:45.0891274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:45.0898548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:45.1808308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:45.5446448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp999k6pcz
2022-09-27T16:46:45.5447578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp999k6pcz/_remote_module_non_scriptable.py
2022-09-27T16:46:46.4571166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23eiio46
2022-09-27T16:46:46.4572232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23eiio46/_remote_module_non_scriptable.py
2022-09-27T16:46:52.2421235Z ok (23.526s)
2022-09-27T16:46:52.2421514Z 
2022-09-27T16:46:52.2422138Z ----------------------------------------------------------------------
2022-09-27T16:46:52.2422496Z Ran 1 test in 23.527s
2022-09-27T16:46:52.2422947Z 
2022-09-27T16:46:52.2423049Z OK
2022-09-27T16:46:52.2423188Z 
2022-09-27T16:46:52.2423305Z Generating XML reports...
2022-09-27T16:46:52.2458821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164628.xml
2022-09-27T16:46:52.9012575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7wx6nv8
2022-09-27T16:46:52.9013756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7wx6nv8/_remote_module_non_scriptable.py
2022-09-27T16:46:52.9315160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xdsh_ek
2022-09-27T16:46:52.9316418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xdsh_ek/_remote_module_non_scriptable.py
2022-09-27T16:46:52.9323771Z [1664297212.932096] [28fd6574525b:121071:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:46:52.9324279Z [1664297212.932096] [28fd6574525b:121071:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:46:52.9364087Z [1664297212.936107] [28fd6574525b:121072:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-09-27T16:46:52.9364579Z [1664297212.936107] [28fd6574525b:121072:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-09-27T16:46:53.4901624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:53.4905348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:46:53.7633636Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:46:53.7638308Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:46:54.0673369Z ok (12.754s)
2022-09-27T16:46:54.0673576Z 
2022-09-27T16:46:54.0674263Z ----------------------------------------------------------------------
2022-09-27T16:46:54.0674622Z Ran 1 test in 12.754s
2022-09-27T16:46:54.0674794Z 
2022-09-27T16:46:54.0674900Z OK
2022-09-27T16:46:54.0675038Z 
2022-09-27T16:46:54.0675173Z Generating XML reports...
2022-09-27T16:46:54.0711906Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164641.xml
2022-09-27T16:46:54.3312736Z ok (13.952s)
2022-09-27T16:46:54.3313211Z 
2022-09-27T16:46:54.3313830Z ----------------------------------------------------------------------
2022-09-27T16:46:54.3314234Z Ran 1 test in 13.953s
2022-09-27T16:46:54.3314411Z 
2022-09-27T16:46:54.3314505Z OK
2022-09-27T16:46:54.3314640Z 
2022-09-27T16:46:54.3314772Z Generating XML reports...
2022-09-27T16:46:54.3350167Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164640.xml
2022-09-27T16:46:54.4592856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:54.4593375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:54.4596237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:54.4596716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:54.9831470Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:46:54.9855288Z 
2022-09-27T16:46:54.9855537Z Running tests...
2022-09-27T16:46:54.9855976Z ----------------------------------------------------------------------
2022-09-27T16:46:56.2122967Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:56.2123621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:56.2125588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:56.2126092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:56.5495991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:56.5496503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:56.5498955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:56.5499433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:56.6698954Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:46:56.6897501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121341
2022-09-27T16:46:56.6904215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121342
2022-09-27T16:46:56.7055435Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:46:56.7078316Z 
2022-09-27T16:46:56.7078611Z Running tests...
2022-09-27T16:46:56.7079092Z ----------------------------------------------------------------------
2022-09-27T16:46:56.7097885Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:46:56.7098250Z 
2022-09-27T16:46:56.7098542Z ----------------------------------------------------------------------
2022-09-27T16:46:56.7098855Z Ran 1 test in 0.002s
2022-09-27T16:46:56.7099028Z 
2022-09-27T16:46:56.7099138Z OK (skipped=1)
2022-09-27T16:46:56.7099296Z 
2022-09-27T16:46:56.7099423Z Generating XML reports...
2022-09-27T16:46:56.7133308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164656.xml
2022-09-27T16:46:57.0450985Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:57.0471989Z 
2022-09-27T16:46:57.0472396Z Running tests...
2022-09-27T16:46:57.0473245Z ----------------------------------------------------------------------
2022-09-27T16:46:57.0491421Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-09-27T16:46:57.0492134Z 
2022-09-27T16:46:57.0492672Z ----------------------------------------------------------------------
2022-09-27T16:46:57.0493283Z Ran 1 test in 0.002s
2022-09-27T16:46:57.0493600Z 
2022-09-27T16:46:57.0493804Z OK (skipped=1)
2022-09-27T16:46:57.0494103Z 
2022-09-27T16:46:57.0494343Z Generating XML reports...
2022-09-27T16:46:57.0527057Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164657.xml
2022-09-27T16:46:58.3855928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:58.3856453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:58.3859151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:58.3859977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:58.4590106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:58.4590606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:58.4592329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:58.4592819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:58.7223563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:58.7224127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:58.7226374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:58.7226876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:58.9841137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:46:58.9842031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:46:58.9842829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:58.9843518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:46:58.9849656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:46:58.9850662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:46:58.9854103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:46:58.9855130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:46:58.9856554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:46:58.9858002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:46:58.9962510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:46:58.9963544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:46:58.9965169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:46:58.9966548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:46:59.0118320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:46:59.0119249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:46:59.0121646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:46:59.0122570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:46:59.2092778Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:46:59.2115097Z 
2022-09-27T16:46:59.2115512Z Running tests...
2022-09-27T16:46:59.2115968Z ----------------------------------------------------------------------
2022-09-27T16:46:59.2134815Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:46:59.2135136Z 
2022-09-27T16:46:59.2135775Z ----------------------------------------------------------------------
2022-09-27T16:46:59.2136507Z Ran 1 test in 0.002s
2022-09-27T16:46:59.2136673Z 
2022-09-27T16:46:59.2136790Z OK (skipped=1)
2022-09-27T16:46:59.2136931Z 
2022-09-27T16:46:59.2137058Z Generating XML reports...
2022-09-27T16:46:59.2170674Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164659.xml
2022-09-27T16:46:59.5539220Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-09-27T16:46:59.5562735Z 
2022-09-27T16:46:59.5563168Z Running tests...
2022-09-27T16:46:59.5563675Z ----------------------------------------------------------------------
2022-09-27T16:46:59.5582540Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-09-27T16:46:59.5582916Z 
2022-09-27T16:46:59.5583202Z ----------------------------------------------------------------------
2022-09-27T16:46:59.5583799Z Ran 1 test in 0.002s
2022-09-27T16:46:59.5583970Z 
2022-09-27T16:46:59.5584081Z OK (skipped=1)
2022-09-27T16:46:59.5584238Z 
2022-09-27T16:46:59.5584366Z Generating XML reports...
2022-09-27T16:46:59.5620488Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164659.xml
2022-09-27T16:47:01.1864704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:01.1865228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:01.1867018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:01.1867519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:01.5769436Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:47:01.5769962Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:47:01.7128104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:01.7150793Z 
2022-09-27T16:47:01.7150940Z Running tests...
2022-09-27T16:47:01.7151693Z ----------------------------------------------------------------------
2022-09-27T16:47:01.7171554Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:01.7171896Z 
2022-09-27T16:47:01.7172190Z ----------------------------------------------------------------------
2022-09-27T16:47:01.7172505Z Ran 1 test in 0.002s
2022-09-27T16:47:01.7172682Z 
2022-09-27T16:47:01.7172796Z OK (skipped=1)
2022-09-27T16:47:01.7172958Z 
2022-09-27T16:47:01.7173314Z Generating XML reports...
2022-09-27T16:47:01.7207940Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164701.xml
2022-09-27T16:47:03.5879893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:03.5880689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:03.5881772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:03.5882249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:04.0799429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:04.0821516Z 
2022-09-27T16:47:04.0821799Z Running tests...
2022-09-27T16:47:04.0822230Z ----------------------------------------------------------------------
2022-09-27T16:47:04.0841249Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:04.0841567Z 
2022-09-27T16:47:04.0841846Z ----------------------------------------------------------------------
2022-09-27T16:47:04.0842449Z Ran 1 test in 0.002s
2022-09-27T16:47:04.0842602Z 
2022-09-27T16:47:04.0842724Z OK (skipped=1)
2022-09-27T16:47:04.0842888Z 
2022-09-27T16:47:04.0843017Z Generating XML reports...
2022-09-27T16:47:04.0875657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164704.xml
2022-09-27T16:47:05.9507498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:05.9507993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:05.9509579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:05.9510092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:06.4481488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:06.4503552Z 
2022-09-27T16:47:06.4503694Z Running tests...
2022-09-27T16:47:06.4504156Z ----------------------------------------------------------------------
2022-09-27T16:47:06.4523397Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:06.4523713Z 
2022-09-27T16:47:06.4523991Z ----------------------------------------------------------------------
2022-09-27T16:47:06.4524336Z Ran 1 test in 0.002s
2022-09-27T16:47:06.4524502Z 
2022-09-27T16:47:06.4524615Z OK (skipped=1)
2022-09-27T16:47:06.4524759Z 
2022-09-27T16:47:06.4524885Z Generating XML reports...
2022-09-27T16:47:06.4558310Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164706.xml
2022-09-27T16:47:08.2887940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:08.2888477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:08.2890205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:08.2890713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:08.7777638Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:08.7799076Z 
2022-09-27T16:47:08.7799314Z Running tests...
2022-09-27T16:47:08.7799937Z ----------------------------------------------------------------------
2022-09-27T16:47:08.7818854Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:08.7819300Z 
2022-09-27T16:47:08.7819593Z ----------------------------------------------------------------------
2022-09-27T16:47:08.7820174Z Ran 1 test in 0.002s
2022-09-27T16:47:08.7820363Z 
2022-09-27T16:47:08.7820476Z OK (skipped=1)
2022-09-27T16:47:08.7820636Z 
2022-09-27T16:47:08.7820765Z Generating XML reports...
2022-09-27T16:47:08.7853667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164708.xml
2022-09-27T16:47:10.5965373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:10.5965904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:10.5966820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:10.5967300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:11.0874500Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:11.0896224Z 
2022-09-27T16:47:11.0896684Z Running tests...
2022-09-27T16:47:11.0897177Z ----------------------------------------------------------------------
2022-09-27T16:47:11.0917120Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:11.0917722Z 
2022-09-27T16:47:11.0918013Z ----------------------------------------------------------------------
2022-09-27T16:47:11.0918331Z Ran 1 test in 0.002s
2022-09-27T16:47:11.0918493Z 
2022-09-27T16:47:11.0918607Z OK (skipped=1)
2022-09-27T16:47:11.0918769Z 
2022-09-27T16:47:11.0918895Z Generating XML reports...
2022-09-27T16:47:11.0951931Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164711.xml
2022-09-27T16:47:12.9161622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:12.9162422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:12.9163603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:12.9164076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:13.4095612Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:13.4117891Z 
2022-09-27T16:47:13.4118175Z Running tests...
2022-09-27T16:47:13.4118594Z ----------------------------------------------------------------------
2022-09-27T16:47:13.4138361Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:13.4138674Z 
2022-09-27T16:47:13.4138963Z ----------------------------------------------------------------------
2022-09-27T16:47:13.4139292Z Ran 1 test in 0.002s
2022-09-27T16:47:13.4139463Z 
2022-09-27T16:47:13.4139558Z OK (skipped=1)
2022-09-27T16:47:13.4139717Z 
2022-09-27T16:47:13.4139841Z Generating XML reports...
2022-09-27T16:47:13.4173019Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164713.xml
2022-09-27T16:47:15.2687490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:15.2688239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:15.2689125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:15.2689620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:15.7764704Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:15.7787553Z 
2022-09-27T16:47:15.7788108Z Running tests...
2022-09-27T16:47:15.7788595Z ----------------------------------------------------------------------
2022-09-27T16:47:15.7808591Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:15.7808961Z 
2022-09-27T16:47:15.7809259Z ----------------------------------------------------------------------
2022-09-27T16:47:15.7809607Z Ran 1 test in 0.002s
2022-09-27T16:47:15.7809773Z 
2022-09-27T16:47:15.7809884Z OK (skipped=1)
2022-09-27T16:47:15.7810023Z 
2022-09-27T16:47:15.7810151Z Generating XML reports...
2022-09-27T16:47:15.7843823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164715.xml
2022-09-27T16:47:17.6797305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:17.6798033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:17.6798627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:17.6799382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:18.1297961Z ok (23.144s)
2022-09-27T16:47:18.1298233Z 
2022-09-27T16:47:18.1298614Z ----------------------------------------------------------------------
2022-09-27T16:47:18.1299232Z Ran 1 test in 23.144s
2022-09-27T16:47:18.1299399Z 
2022-09-27T16:47:18.1299491Z OK
2022-09-27T16:47:18.1299612Z 
2022-09-27T16:47:18.1299746Z Generating XML reports...
2022-09-27T16:47:18.1334744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164654.xml
2022-09-27T16:47:18.2417697Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:18.2439618Z 
2022-09-27T16:47:18.2439864Z Running tests...
2022-09-27T16:47:18.2440699Z ----------------------------------------------------------------------
2022-09-27T16:47:18.2460275Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:18.2460744Z 
2022-09-27T16:47:18.2461052Z ----------------------------------------------------------------------
2022-09-27T16:47:18.2461397Z Ran 1 test in 0.002s
2022-09-27T16:47:18.2461622Z 
2022-09-27T16:47:18.2461820Z OK (skipped=1)
2022-09-27T16:47:18.2462083Z 
2022-09-27T16:47:18.2462210Z Generating XML reports...
2022-09-27T16:47:18.2497173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164718.xml
2022-09-27T16:47:20.1355847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:20.1356357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:20.1357202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:20.1357691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:20.1884634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:20.1885096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:20.1887746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:20.1888250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:20.6352280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:20.6374300Z 
2022-09-27T16:47:20.6374482Z Running tests...
2022-09-27T16:47:20.6374924Z ----------------------------------------------------------------------
2022-09-27T16:47:20.6734386Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:47:20.6757477Z 
2022-09-27T16:47:20.6757694Z Running tests...
2022-09-27T16:47:20.6758417Z ----------------------------------------------------------------------
2022-09-27T16:47:22.2690816Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:22.2783705Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:22.2886058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121832
2022-09-27T16:47:22.2892899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121833
2022-09-27T16:47:22.2975734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121835
2022-09-27T16:47:22.2981963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121836
2022-09-27T16:47:23.8791719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:23.8792350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:23.8793569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:23.8794047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:23.8843221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:23.8844023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:23.8847190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:23.8847670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:23.8997040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:23.8997507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:23.9001186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:23.9001708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:23.9326434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:23.9326980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:23.9330146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:23.9330915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:24.3910406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:24.4311541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:24.4312454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:24.4315820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:24.4322363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:24.4892981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:24.4893768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:24.4894544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:24.4895497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:24.4901787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:24.4902782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:24.5326049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:24.9046945Z skip: Skipped due to small world size. (4.228s)
2022-09-27T16:47:24.9047441Z 
2022-09-27T16:47:24.9048061Z ----------------------------------------------------------------------
2022-09-27T16:47:24.9048675Z Ran 1 test in 4.229s
2022-09-27T16:47:24.9048984Z 
2022-09-27T16:47:24.9049183Z OK (skipped=1)
2022-09-27T16:47:24.9049481Z 
2022-09-27T16:47:24.9049729Z Generating XML reports...
2022-09-27T16:47:24.9087176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164720.xml
2022-09-27T16:47:26.0689681Z STAGE:2022-09-27 16:47:26 121832:121832 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:26.1858099Z STAGE:2022-09-27 16:47:26 121833:121833 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:26.2555331Z STAGE:2022-09-27 16:47:26 121833:121833 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:26.2556148Z STAGE:2022-09-27 16:47:26 121832:121832 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:26.4452610Z STAGE:2022-09-27 16:47:26 121832:121832 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:26.4453928Z STAGE:2022-09-27 16:47:26 121833:121833 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:26.4466981Z STAGE:2022-09-27 16:47:26 121833:121833 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:26.4468155Z STAGE:2022-09-27 16:47:26 121832:121832 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:26.8995236Z ok (6.262s)
2022-09-27T16:47:26.8995602Z 
2022-09-27T16:47:26.8996262Z ----------------------------------------------------------------------
2022-09-27T16:47:26.8996889Z Ran 1 test in 6.262s
2022-09-27T16:47:26.8997184Z 
2022-09-27T16:47:26.8997322Z OK
2022-09-27T16:47:26.8997591Z 
2022-09-27T16:47:26.8997832Z Generating XML reports...
2022-09-27T16:47:26.9036361Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164720.xml
2022-09-27T16:47:27.0619782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:27.0620741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:27.0621865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:27.0622792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:27.5688822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:47:27.5710868Z 
2022-09-27T16:47:27.5711324Z Running tests...
2022-09-27T16:47:27.5711845Z ----------------------------------------------------------------------
2022-09-27T16:47:28.9386202Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:28.9386815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:28.9387876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:28.9388347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:29.1396746Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:29.1587756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122060
2022-09-27T16:47:29.1594418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122061
2022-09-27T16:47:29.4538459Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:29.4561516Z 
2022-09-27T16:47:29.4562252Z Running tests...
2022-09-27T16:47:29.4562780Z ----------------------------------------------------------------------
2022-09-27T16:47:29.4583145Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:29.4583484Z 
2022-09-27T16:47:29.4583757Z ----------------------------------------------------------------------
2022-09-27T16:47:29.4584094Z Ran 1 test in 0.002s
2022-09-27T16:47:29.4584261Z 
2022-09-27T16:47:29.4584371Z OK (skipped=1)
2022-09-27T16:47:29.4584531Z 
2022-09-27T16:47:29.4584659Z Generating XML reports...
2022-09-27T16:47:29.4619183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164729.xml
2022-09-27T16:47:30.7706190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:30.7706712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:30.7707981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:30.7708458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:30.8732080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:30.8732562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:30.8734338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:30.8734832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:31.3875835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:31.3876351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:31.3878845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:31.3879353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:31.3885959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:31.3886493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:31.3887192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:31.3887880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:31.3995245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:31.4900604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:31.8647221Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:31.8670127Z 
2022-09-27T16:47:31.8670443Z Running tests...
2022-09-27T16:47:31.8670891Z ----------------------------------------------------------------------
2022-09-27T16:47:33.3976158Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:33.4166424Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122169
2022-09-27T16:47:33.4172959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122170
2022-09-27T16:47:33.5023907Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-09-27T16:47:33.9711135Z ok (6.400s)
2022-09-27T16:47:33.9711346Z 
2022-09-27T16:47:33.9712001Z ----------------------------------------------------------------------
2022-09-27T16:47:33.9712360Z Ran 1 test in 6.400s
2022-09-27T16:47:33.9712521Z 
2022-09-27T16:47:33.9712621Z OK
2022-09-27T16:47:33.9712771Z 
2022-09-27T16:47:33.9713345Z Generating XML reports...
2022-09-27T16:47:33.9747977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164727.xml
2022-09-27T16:47:35.0679197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:35.0681744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:35.0683674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:35.0684829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:35.0781615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:35.0783287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:35.0790586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:35.0791742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:35.6102915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:35.6297856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:35.6298707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:35.6305862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:35.6312597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:35.7311832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:36.0387777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:36.0388282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:36.0389122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:36.0389618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:36.5505805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:47:36.5528589Z 
2022-09-27T16:47:36.5528844Z Running tests...
2022-09-27T16:47:36.5529300Z ----------------------------------------------------------------------
2022-09-27T16:47:38.1539523Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:38.1735841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122286
2022-09-27T16:47:38.1741454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122287
2022-09-27T16:47:38.7287212Z ok (6.861s)
2022-09-27T16:47:38.7287469Z 
2022-09-27T16:47:38.7287883Z ----------------------------------------------------------------------
2022-09-27T16:47:38.7288293Z Ran 1 test in 6.862s
2022-09-27T16:47:38.7288465Z 
2022-09-27T16:47:38.7288561Z OK
2022-09-27T16:47:38.7288682Z 
2022-09-27T16:47:38.7288820Z Generating XML reports...
2022-09-27T16:47:38.7323151Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164731.xml
2022-09-27T16:47:39.8569973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:39.8571688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:39.8574708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:39.8576873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:39.8983487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:39.8984579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:39.8990160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:39.8991292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:40.4315028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:40.4315599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:40.4316384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:40.4317117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:40.4322705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:40.4324127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:40.4430846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:47:40.4431366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:47:40.4432062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:47:40.4432762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:47:40.4435405Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-09-27T16:47:40.4435846Z No ranks successfully processed in monitoredBarrier.
2022-09-27T16:47:40.4465141Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-09-27T16:47:40.8267320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:40.8267826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:40.8268867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:40.8269355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:40.8808061Z ok (4.328s)
2022-09-27T16:47:40.8808236Z 
2022-09-27T16:47:40.8809384Z ----------------------------------------------------------------------
2022-09-27T16:47:40.8810170Z Ran 1 test in 4.328s
2022-09-27T16:47:40.8810449Z 
2022-09-27T16:47:40.8810547Z OK
2022-09-27T16:47:40.8810686Z 
2022-09-27T16:47:40.8810812Z Generating XML reports...
2022-09-27T16:47:40.8846643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164736.xml
2022-09-27T16:47:41.3317923Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:41.3340976Z 
2022-09-27T16:47:41.3341500Z Running tests...
2022-09-27T16:47:41.3342456Z ----------------------------------------------------------------------
2022-09-27T16:47:41.3362153Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:47:41.3362777Z 
2022-09-27T16:47:41.3363409Z ----------------------------------------------------------------------
2022-09-27T16:47:41.3364091Z Ran 1 test in 0.002s
2022-09-27T16:47:41.3364272Z 
2022-09-27T16:47:41.3364366Z OK (skipped=1)
2022-09-27T16:47:41.3364527Z 
2022-09-27T16:47:41.3364654Z Generating XML reports...
2022-09-27T16:47:41.3399178Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164741.xml
2022-09-27T16:47:42.9730103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:42.9730772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:42.9731984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:42.9732794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:43.3461426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:43.3462259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:43.3463531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:43.3464432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:43.4545956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:47:43.4568334Z 
2022-09-27T16:47:43.4569071Z Running tests...
2022-09-27T16:47:43.4569941Z ----------------------------------------------------------------------
2022-09-27T16:47:43.8439295Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:43.8462052Z 
2022-09-27T16:47:43.8462359Z Running tests...
2022-09-27T16:47:43.8463337Z ----------------------------------------------------------------------
2022-09-27T16:47:45.0408344Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:45.0603828Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122468
2022-09-27T16:47:45.0610278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122469
2022-09-27T16:47:45.4057692Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:45.4257057Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122471
2022-09-27T16:47:45.4263726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122472
2022-09-27T16:47:46.7672525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:46.7673082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:46.7674145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:46.7674643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:46.7791027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:46.7791524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:46.7795689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:46.7796185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:47.1001272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:47.1001781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:47.1003357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:47.1003830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:47.1240830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:47.1241325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:47.1245080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:47.1245613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:47.2932492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:47.2933052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:47.2933848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:47.2934560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:47.3040472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:47.3942872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:47.4150028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:47:47.4150663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:47:47.4151604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:47:47.4152309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:47:47.5257675Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-09-27T16:47:47.6006587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:47.6275438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:47.6276502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:47.6310877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:47.6317754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:47.7290302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:47.9679273Z ok (4.511s)
2022-09-27T16:47:47.9679519Z 
2022-09-27T16:47:47.9679927Z ----------------------------------------------------------------------
2022-09-27T16:47:47.9680880Z Ran 1 test in 4.511s
2022-09-27T16:47:47.9681060Z 
2022-09-27T16:47:47.9681157Z OK
2022-09-27T16:47:47.9681278Z 
2022-09-27T16:47:47.9681418Z Generating XML reports...
2022-09-27T16:47:47.9716969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164743.xml
2022-09-27T16:47:49.2579719Z STAGE:2022-09-27 16:47:49 122472:122472 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:49.3782996Z STAGE:2022-09-27 16:47:49 122471:122471 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:49.4687917Z STAGE:2022-09-27 16:47:49 122471:122471 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:49.4688568Z STAGE:2022-09-27 16:47:49 122472:122472 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:49.6441201Z STAGE:2022-09-27 16:47:49 122471:122471 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:49.6442751Z STAGE:2022-09-27 16:47:49 122472:122472 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:49.6456067Z STAGE:2022-09-27 16:47:49 122471:122471 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:49.6458855Z STAGE:2022-09-27 16:47:49 122472:122472 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:50.1521245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:50.1521782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:50.1522864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:50.1523353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:50.2381920Z ok (6.392s)
2022-09-27T16:47:50.2382145Z 
2022-09-27T16:47:50.2382478Z ----------------------------------------------------------------------
2022-09-27T16:47:50.2383027Z Ran 1 test in 6.392s
2022-09-27T16:47:50.2383325Z 
2022-09-27T16:47:50.2383493Z OK
2022-09-27T16:47:50.2383731Z 
2022-09-27T16:47:50.2383987Z Generating XML reports...
2022-09-27T16:47:50.2420783Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164743.xml
2022-09-27T16:47:50.6843776Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:47:50.6866907Z 
2022-09-27T16:47:50.6867385Z Running tests...
2022-09-27T16:47:50.6868518Z ----------------------------------------------------------------------
2022-09-27T16:47:52.2956898Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:52.2957544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:52.2960427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:52.2961184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:52.3014607Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:52.3216969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122702
2022-09-27T16:47:52.3223067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122703
2022-09-27T16:47:52.7871280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:47:52.7894595Z 
2022-09-27T16:47:52.7894874Z Running tests...
2022-09-27T16:47:52.7895399Z ----------------------------------------------------------------------
2022-09-27T16:47:53.9178952Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:53.9179560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:53.9180474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:53.9180971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:53.9511733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:53.9512231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:53.9514468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:53.9514960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:54.3745467Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:47:54.3942117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122770
2022-09-27T16:47:54.3949185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122771
2022-09-27T16:47:54.4833283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:54.4833824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:54.4834837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:54.4835575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:54.4842067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:54.4842546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:54.8285970Z skip: Skipped due to small world size. (4.141s)
2022-09-27T16:47:54.8286285Z 
2022-09-27T16:47:54.8286685Z ----------------------------------------------------------------------
2022-09-27T16:47:54.8287027Z Ran 1 test in 4.142s
2022-09-27T16:47:54.8287176Z 
2022-09-27T16:47:54.8287286Z OK (skipped=1)
2022-09-27T16:47:54.8287440Z 
2022-09-27T16:47:54.8287570Z Generating XML reports...
2022-09-27T16:47:54.8324789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164750.xml
2022-09-27T16:47:56.0081297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:56.0082692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:56.0084273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:56.0085254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:56.0378951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:56.0379919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:56.0383497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:56.0384526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:56.5033563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:47:56.5480145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:47:56.5481265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:56.5539794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:47:56.5546738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:47:56.6494699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:47:56.8938437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:56.8939417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:56.8941270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:56.8942204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:57.4039020Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:47:57.4063504Z 
2022-09-27T16:47:57.4063905Z Running tests...
2022-09-27T16:47:57.4064419Z ----------------------------------------------------------------------
2022-09-27T16:47:57.4089911Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:47:57.4090251Z 
2022-09-27T16:47:57.4090675Z ----------------------------------------------------------------------
2022-09-27T16:47:57.4090996Z Ran 1 test in 0.003s
2022-09-27T16:47:57.4091166Z 
2022-09-27T16:47:57.4091284Z OK (skipped=1)
2022-09-27T16:47:57.4091447Z 
2022-09-27T16:47:57.4091875Z Generating XML reports...
2022-09-27T16:47:57.4126946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164757.xml
2022-09-27T16:47:58.1535267Z STAGE:2022-09-27 16:47:58 122771:122771 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:58.2442448Z STAGE:2022-09-27 16:47:58 122770:122770 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:58.3346741Z STAGE:2022-09-27 16:47:58 122770:122770 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:58.3347896Z STAGE:2022-09-27 16:47:58 122771:122771 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:58.5705856Z STAGE:2022-09-27 16:47:58 122770:122770 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:58.5713196Z STAGE:2022-09-27 16:47:58 122771:122771 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:47:58.5721956Z STAGE:2022-09-27 16:47:58 122770:122770 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:58.5730813Z STAGE:2022-09-27 16:47:58 122771:122771 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:47:59.2066854Z ok (6.417s)
2022-09-27T16:47:59.2067068Z 
2022-09-27T16:47:59.2067486Z ----------------------------------------------------------------------
2022-09-27T16:47:59.2067831Z Ran 1 test in 6.417s
2022-09-27T16:47:59.2067980Z 
2022-09-27T16:47:59.2068080Z OK
2022-09-27T16:47:59.2068217Z 
2022-09-27T16:47:59.2068351Z Generating XML reports...
2022-09-27T16:47:59.2104349Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164752.xml
2022-09-27T16:47:59.3479720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:47:59.3480940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:47:59.3482392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:47:59.3483327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:47:59.8495813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:47:59.8518195Z 
2022-09-27T16:47:59.8518496Z Running tests...
2022-09-27T16:47:59.8518934Z ----------------------------------------------------------------------
2022-09-27T16:47:59.8544646Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:47:59.8545162Z 
2022-09-27T16:47:59.8545423Z ----------------------------------------------------------------------
2022-09-27T16:47:59.8545760Z Ran 1 test in 0.003s
2022-09-27T16:47:59.8545925Z 
2022-09-27T16:47:59.8546036Z OK (skipped=1)
2022-09-27T16:47:59.8546196Z 
2022-09-27T16:47:59.8546322Z Generating XML reports...
2022-09-27T16:47:59.8579385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164759.xml
2022-09-27T16:48:01.2151975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:01.2153062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:01.2154671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:01.2155644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:01.6940811Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:01.6963837Z 
2022-09-27T16:48:01.6964281Z Running tests...
2022-09-27T16:48:01.6964788Z ----------------------------------------------------------------------
2022-09-27T16:48:01.6985018Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:48:01.6985400Z 
2022-09-27T16:48:01.6985726Z ----------------------------------------------------------------------
2022-09-27T16:48:01.6986053Z Ran 1 test in 0.002s
2022-09-27T16:48:01.6986232Z 
2022-09-27T16:48:01.6986344Z OK (skipped=1)
2022-09-27T16:48:01.6986502Z 
2022-09-27T16:48:01.6986629Z Generating XML reports...
2022-09-27T16:48:01.7021318Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164801.xml
2022-09-27T16:48:01.8099538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:01.8100064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:01.8101567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:01.8102058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:02.3144822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:02.3168721Z 
2022-09-27T16:48:02.3169274Z Running tests...
2022-09-27T16:48:02.3170114Z ----------------------------------------------------------------------
2022-09-27T16:48:02.3192850Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-09-27T16:48:02.3193215Z 
2022-09-27T16:48:02.3205452Z ----------------------------------------------------------------------
2022-09-27T16:48:02.3205893Z Ran 1 test in 0.002s
2022-09-27T16:48:02.3206070Z 
2022-09-27T16:48:02.3206192Z OK (skipped=1)
2022-09-27T16:48:02.3206353Z 
2022-09-27T16:48:02.3206496Z Generating XML reports...
2022-09-27T16:48:02.3229192Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164802.xml
2022-09-27T16:48:03.5452496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:03.5453533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:03.5455209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:03.5456187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:04.0293172Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:04.0314460Z 
2022-09-27T16:48:04.0315136Z Running tests...
2022-09-27T16:48:04.0316013Z ----------------------------------------------------------------------
2022-09-27T16:48:04.0334116Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:48:04.0334748Z 
2022-09-27T16:48:04.0335288Z ----------------------------------------------------------------------
2022-09-27T16:48:04.0335944Z Ran 1 test in 0.002s
2022-09-27T16:48:04.0336286Z 
2022-09-27T16:48:04.0336501Z OK (skipped=1)
2022-09-27T16:48:04.0336774Z 
2022-09-27T16:48:04.0337014Z Generating XML reports...
2022-09-27T16:48:04.0369530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164804.xml
2022-09-27T16:48:04.2556070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:04.2557088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:04.2558787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:04.2559723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:04.7494545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:04.7518950Z 
2022-09-27T16:48:04.7519446Z Running tests...
2022-09-27T16:48:04.7520529Z ----------------------------------------------------------------------
2022-09-27T16:48:04.7547783Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-09-27T16:48:04.7548126Z 
2022-09-27T16:48:04.7548844Z ----------------------------------------------------------------------
2022-09-27T16:48:04.7549536Z Ran 1 test in 0.003s
2022-09-27T16:48:04.7549863Z 
2022-09-27T16:48:04.7550057Z OK (skipped=1)
2022-09-27T16:48:04.7550331Z 
2022-09-27T16:48:04.7550572Z Generating XML reports...
2022-09-27T16:48:04.7586842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164804.xml
2022-09-27T16:48:05.9399842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:05.9400796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:05.9402217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:05.9402710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:06.4285326Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:06.4308311Z 
2022-09-27T16:48:06.4308786Z Running tests...
2022-09-27T16:48:06.4309325Z ----------------------------------------------------------------------
2022-09-27T16:48:06.4333213Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:48:06.4333702Z 
2022-09-27T16:48:06.4334007Z ----------------------------------------------------------------------
2022-09-27T16:48:06.4334500Z Ran 1 test in 0.003s
2022-09-27T16:48:06.4334795Z 
2022-09-27T16:48:06.4334916Z OK (skipped=1)
2022-09-27T16:48:06.4335076Z 
2022-09-27T16:48:06.4335203Z Generating XML reports...
2022-09-27T16:48:06.4370462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164806.xml
2022-09-27T16:48:06.7387422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:06.7387982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:06.7390047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:06.7390556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:07.2425275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:07.2449857Z 
2022-09-27T16:48:07.2450342Z Running tests...
2022-09-27T16:48:07.2450877Z ----------------------------------------------------------------------
2022-09-27T16:48:07.2474207Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-09-27T16:48:07.2474577Z 
2022-09-27T16:48:07.2474876Z ----------------------------------------------------------------------
2022-09-27T16:48:07.2475200Z Ran 1 test in 0.002s
2022-09-27T16:48:07.2475377Z 
2022-09-27T16:48:07.2475491Z OK (skipped=1)
2022-09-27T16:48:07.2475649Z 
2022-09-27T16:48:07.2475779Z Generating XML reports...
2022-09-27T16:48:07.2511179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164807.xml
2022-09-27T16:48:08.3135974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:08.3136515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:08.3139215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:08.3139718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:08.8064605Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:08.8088747Z 
2022-09-27T16:48:08.8089067Z Running tests...
2022-09-27T16:48:08.8089542Z ----------------------------------------------------------------------
2022-09-27T16:48:08.8109236Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:48:08.8109565Z 
2022-09-27T16:48:08.8110592Z ----------------------------------------------------------------------
2022-09-27T16:48:08.8111254Z Ran 1 test in 0.002s
2022-09-27T16:48:08.8111562Z 
2022-09-27T16:48:08.8111786Z OK (skipped=1)
2022-09-27T16:48:08.8112058Z 
2022-09-27T16:48:08.8112303Z Generating XML reports...
2022-09-27T16:48:08.8148621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164808.xml
2022-09-27T16:48:09.1726865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:09.1727415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:09.1728491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:09.1729312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:09.6601230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:09.6623424Z 
2022-09-27T16:48:09.6623688Z Running tests...
2022-09-27T16:48:09.6624130Z ----------------------------------------------------------------------
2022-09-27T16:48:09.6645132Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:48:09.6645446Z 
2022-09-27T16:48:09.6645726Z ----------------------------------------------------------------------
2022-09-27T16:48:09.6646066Z Ran 1 test in 0.002s
2022-09-27T16:48:09.6646234Z 
2022-09-27T16:48:09.6646366Z OK (skipped=1)
2022-09-27T16:48:09.6646509Z 
2022-09-27T16:48:09.6646640Z Generating XML reports...
2022-09-27T16:48:09.6679558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164809.xml
2022-09-27T16:48:10.7401969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:10.7402489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:10.7404728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:10.7405232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:11.2340889Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:11.2364563Z 
2022-09-27T16:48:11.2365003Z Running tests...
2022-09-27T16:48:11.2365526Z ----------------------------------------------------------------------
2022-09-27T16:48:11.5291995Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:11.5292608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:11.5293462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:11.5293936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:12.0219811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:12.0242418Z 
2022-09-27T16:48:12.0242843Z Running tests...
2022-09-27T16:48:12.0243311Z ----------------------------------------------------------------------
2022-09-27T16:48:12.0266959Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:48:12.0267314Z 
2022-09-27T16:48:12.0267587Z ----------------------------------------------------------------------
2022-09-27T16:48:12.0267924Z Ran 1 test in 0.003s
2022-09-27T16:48:12.0268102Z 
2022-09-27T16:48:12.0268212Z OK (skipped=1)
2022-09-27T16:48:12.0268372Z 
2022-09-27T16:48:12.0268499Z Generating XML reports...
2022-09-27T16:48:12.0302450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164812.xml
2022-09-27T16:48:12.8251369Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:12.8450352Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123262
2022-09-27T16:48:12.8456939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123263
2022-09-27T16:48:14.0352029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:14.0352567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:14.0353985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:14.0354752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:14.4519345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:14.4519848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:14.4521539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:14.4522031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:14.5084738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:14.5085215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:14.5088798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:14.5089292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:14.5252767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:14.5275854Z 
2022-09-27T16:48:14.5276325Z Running tests...
2022-09-27T16:48:14.5276851Z ----------------------------------------------------------------------
2022-09-27T16:48:14.5298265Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:48:14.5298646Z 
2022-09-27T16:48:14.5298941Z ----------------------------------------------------------------------
2022-09-27T16:48:14.5299276Z Ran 1 test in 0.002s
2022-09-27T16:48:14.5299446Z 
2022-09-27T16:48:14.5299563Z OK (skipped=1)
2022-09-27T16:48:14.5299726Z 
2022-09-27T16:48:14.5299852Z Generating XML reports...
2022-09-27T16:48:14.5334826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164814.xml
2022-09-27T16:48:14.9607004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:15.0244530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:15.0245357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:15.0315991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:15.0322522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:15.1258260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:16.4441828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:16.4442582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:16.4443228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:16.4443712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:16.6335297Z STAGE:2022-09-27 16:48:16 123263:123263 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:48:16.7544732Z STAGE:2022-09-27 16:48:16 123262:123262 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:48:16.9482769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:16.9505266Z 
2022-09-27T16:48:16.9505514Z Running tests...
2022-09-27T16:48:16.9505969Z ----------------------------------------------------------------------
2022-09-27T16:48:18.0514490Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... STAGE:2022-09-27 16:48:18 123262:123262 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:48:18.0542334Z STAGE:2022-09-27 16:48:18 123263:123263 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:48:18.2252746Z STAGE:2022-09-27 16:48:18 123263:123263 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:48:18.2253364Z STAGE:2022-09-27 16:48:18 123262:123262 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:48:18.2365294Z STAGE:2022-09-27 16:48:18 123263:123263 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:48:18.2393655Z STAGE:2022-09-27 16:48:18 123262:123262 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:48:18.5561094Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:18.5769979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123420
2022-09-27T16:48:18.5774973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123421
2022-09-27T16:48:19.0592054Z ok (7.822s)
2022-09-27T16:48:19.0592274Z 
2022-09-27T16:48:19.0592673Z ----------------------------------------------------------------------
2022-09-27T16:48:19.0593022Z Ran 1 test in 7.823s
2022-09-27T16:48:19.0593170Z 
2022-09-27T16:48:19.0593563Z OK
2022-09-27T16:48:19.0593705Z 
2022-09-27T16:48:19.0594004Z Generating XML reports...
2022-09-27T16:48:19.0629331Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164811.xml
2022-09-27T16:48:20.2600113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:20.2600971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:20.2606708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:20.2607527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:20.2755918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:20.2756623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:20.2759829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:20.2760728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:20.8004348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:20.8004951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:20.8005766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:20.8006715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:20.8013493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:20.8013990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:21.1758880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:21.1759393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:21.1762041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:21.1762548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:21.1836865Z ok (4.233s)
2022-09-27T16:48:21.1837304Z 
2022-09-27T16:48:21.1837939Z ----------------------------------------------------------------------
2022-09-27T16:48:21.1838320Z Ran 1 test in 4.233s
2022-09-27T16:48:21.1838488Z 
2022-09-27T16:48:21.1838591Z OK
2022-09-27T16:48:21.1838726Z 
2022-09-27T16:48:21.1838840Z Generating XML reports...
2022-09-27T16:48:21.1875265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164816.xml
2022-09-27T16:48:21.6754543Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:21.6776386Z 
2022-09-27T16:48:21.6776810Z Running tests...
2022-09-27T16:48:21.6777305Z ----------------------------------------------------------------------
2022-09-27T16:48:23.2380071Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:23.2380664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:23.2382136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:23.2382627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:23.2540387Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:23.2733365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123562
2022-09-27T16:48:23.2739481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123563
2022-09-27T16:48:23.7341654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:23.7365347Z 
2022-09-27T16:48:23.7365595Z Running tests...
2022-09-27T16:48:23.7366037Z ----------------------------------------------------------------------
2022-09-27T16:48:24.8880830Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:24.8881490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:24.8882296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:24.8882783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:24.9186228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:24.9186715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:24.9189150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:24.9189621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:25.3421465Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:25.3617930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123630
2022-09-27T16:48:25.3624695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123631
2022-09-27T16:48:25.4182587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:25.4246558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:25.4247377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:25.4284113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:25.4290739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:25.5261659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:27.0978940Z STAGE:2022-09-27 16:48:27 123563:123563 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:48:27.1232490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:27.1233460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:27.1235056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:27.1235943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:27.1331187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:27.1332156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:27.1333837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:27.1334747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:27.1899614Z STAGE:2022-09-27 16:48:27 123562:123562 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:48:27.6881369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:27.6882275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:27.6883502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:27.6884614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:27.6989792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:27.7895428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:28.1694654Z ok (4.433s)
2022-09-27T16:48:28.1694946Z 
2022-09-27T16:48:28.1695528Z ----------------------------------------------------------------------
2022-09-27T16:48:28.1696051Z Ran 1 test in 4.433s
2022-09-27T16:48:28.1696302Z 
2022-09-27T16:48:28.1696441Z OK
2022-09-27T16:48:28.1696645Z 
2022-09-27T16:48:28.1696820Z Generating XML reports...
2022-09-27T16:48:28.1847995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164823.xml
2022-09-27T16:48:28.5016939Z STAGE:2022-09-27 16:48:28 123562:123562 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:48:28.5045921Z STAGE:2022-09-27 16:48:28 123563:123563 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:48:28.6662895Z STAGE:2022-09-27 16:48:28 123563:123563 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:48:28.7081224Z STAGE:2022-09-27 16:48:28 123562:123562 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:48:28.7197881Z STAGE:2022-09-27 16:48:28 123563:123563 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:48:28.7225081Z STAGE:2022-09-27 16:48:28 123562:123562 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:48:29.4869877Z ok (7.809s)
2022-09-27T16:48:29.4870147Z 
2022-09-27T16:48:29.4870535Z ----------------------------------------------------------------------
2022-09-27T16:48:29.4870880Z Ran 1 test in 7.809s
2022-09-27T16:48:29.4871050Z 
2022-09-27T16:48:29.4871153Z OK
2022-09-27T16:48:29.4871271Z 
2022-09-27T16:48:29.4871409Z Generating XML reports...
2022-09-27T16:48:29.4905785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164821.xml
2022-09-27T16:48:30.3283844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:30.3284381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:30.3286281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:30.3286776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:30.9046908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:30.9068756Z 
2022-09-27T16:48:30.9069213Z Running tests...
2022-09-27T16:48:30.9069740Z ----------------------------------------------------------------------
2022-09-27T16:48:30.9088982Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:48:30.9089333Z 
2022-09-27T16:48:30.9089804Z ----------------------------------------------------------------------
2022-09-27T16:48:30.9090261Z Ran 1 test in 0.002s
2022-09-27T16:48:30.9090428Z 
2022-09-27T16:48:30.9090540Z OK (skipped=1)
2022-09-27T16:48:30.9090698Z 
2022-09-27T16:48:30.9090833Z Generating XML reports...
2022-09-27T16:48:30.9123557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164830.xml
2022-09-27T16:48:31.5273905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:31.5274442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:31.5275713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:31.5276201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:32.0835494Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:32.0858894Z 
2022-09-27T16:48:32.0859307Z Running tests...
2022-09-27T16:48:32.0859820Z ----------------------------------------------------------------------
2022-09-27T16:48:32.0878184Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:48:32.0878501Z 
2022-09-27T16:48:32.0878806Z ----------------------------------------------------------------------
2022-09-27T16:48:32.0879839Z Ran 1 test in 0.002s
2022-09-27T16:48:32.0880009Z 
2022-09-27T16:48:32.0880337Z OK (skipped=1)
2022-09-27T16:48:32.0880682Z 
2022-09-27T16:48:32.0881133Z Generating XML reports...
2022-09-27T16:48:32.0914387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164832.xml
2022-09-27T16:48:32.7619582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:32.7620100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:32.7621441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:32.7621930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:33.2989736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:33.3011793Z 
2022-09-27T16:48:33.3012096Z Running tests...
2022-09-27T16:48:33.3012519Z ----------------------------------------------------------------------
2022-09-27T16:48:33.3029911Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-09-27T16:48:33.3030435Z 
2022-09-27T16:48:33.3030879Z ----------------------------------------------------------------------
2022-09-27T16:48:33.3031228Z Ran 1 test in 0.002s
2022-09-27T16:48:33.3031404Z 
2022-09-27T16:48:33.3031528Z OK (skipped=1)
2022-09-27T16:48:33.3031688Z 
2022-09-27T16:48:33.3031818Z Generating XML reports...
2022-09-27T16:48:33.3065947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164833.xml
2022-09-27T16:48:33.9911612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:33.9912127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:33.9914301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:33.9915091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:34.4924471Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:34.4947797Z 
2022-09-27T16:48:34.4948225Z Running tests...
2022-09-27T16:48:34.4948690Z ----------------------------------------------------------------------
2022-09-27T16:48:34.4967416Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-09-27T16:48:34.4968418Z 
2022-09-27T16:48:34.4968705Z ----------------------------------------------------------------------
2022-09-27T16:48:34.4969056Z Ran 1 test in 0.002s
2022-09-27T16:48:34.4969223Z 
2022-09-27T16:48:34.4969334Z OK (skipped=1)
2022-09-27T16:48:34.4969499Z 
2022-09-27T16:48:34.4969627Z Generating XML reports...
2022-09-27T16:48:34.5005919Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164834.xml
2022-09-27T16:48:35.1229467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:35.1230478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:35.1231655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:35.1232598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:35.6616396Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:35.6639242Z 
2022-09-27T16:48:35.6639384Z Running tests...
2022-09-27T16:48:35.6640982Z ----------------------------------------------------------------------
2022-09-27T16:48:36.4155199Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:36.4156353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:36.4157498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:36.4158389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:36.9209455Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:36.9233266Z 
2022-09-27T16:48:36.9233683Z Running tests...
2022-09-27T16:48:36.9234170Z ----------------------------------------------------------------------
2022-09-27T16:48:36.9256762Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-09-27T16:48:36.9257429Z 
2022-09-27T16:48:36.9257723Z ----------------------------------------------------------------------
2022-09-27T16:48:36.9258053Z Ran 1 test in 0.002s
2022-09-27T16:48:36.9258238Z 
2022-09-27T16:48:36.9258347Z OK (skipped=1)
2022-09-27T16:48:36.9258508Z 
2022-09-27T16:48:36.9258641Z Generating XML reports...
2022-09-27T16:48:36.9294254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164836.xml
2022-09-27T16:48:37.2197680Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:37.2363492Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.572s)
2022-09-27T16:48:37.2364081Z 
2022-09-27T16:48:37.2364380Z ----------------------------------------------------------------------
2022-09-27T16:48:37.2364724Z Ran 1 test in 1.572s
2022-09-27T16:48:37.2364893Z 
2022-09-27T16:48:37.2365005Z OK (skipped=1)
2022-09-27T16:48:37.2365146Z 
2022-09-27T16:48:37.2365551Z Generating XML reports...
2022-09-27T16:48:37.2397484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164835.xml
2022-09-27T16:48:38.9434409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:38.9434927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:38.9436482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:38.9436993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:39.3386118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:39.3386699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:39.3388852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:39.3389364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:39.4462523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:39.4486466Z 
2022-09-27T16:48:39.4486901Z Running tests...
2022-09-27T16:48:39.4487418Z ----------------------------------------------------------------------
2022-09-27T16:48:39.4505202Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:48:39.4505523Z 
2022-09-27T16:48:39.4505813Z ----------------------------------------------------------------------
2022-09-27T16:48:39.4506129Z Ran 1 test in 0.002s
2022-09-27T16:48:39.4506297Z 
2022-09-27T16:48:39.4506417Z OK (skipped=1)
2022-09-27T16:48:39.4506602Z 
2022-09-27T16:48:39.4506736Z Generating XML reports...
2022-09-27T16:48:39.4541930Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164839.xml
2022-09-27T16:48:39.8212137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:39.8234513Z 
2022-09-27T16:48:39.8234812Z Running tests...
2022-09-27T16:48:39.8235253Z ----------------------------------------------------------------------
2022-09-27T16:48:41.3047509Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:41.3048574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:41.3049665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:41.3050980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:41.4025326Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:41.4224051Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124029
2022-09-27T16:48:41.4229870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124030
2022-09-27T16:48:41.7968636Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:41.7991939Z 
2022-09-27T16:48:41.7992461Z Running tests...
2022-09-27T16:48:41.7992943Z ----------------------------------------------------------------------
2022-09-27T16:48:41.8010566Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-09-27T16:48:41.8010911Z 
2022-09-27T16:48:41.8011208Z ----------------------------------------------------------------------
2022-09-27T16:48:41.8011527Z Ran 1 test in 0.002s
2022-09-27T16:48:41.8011716Z 
2022-09-27T16:48:41.8011830Z OK (skipped=1)
2022-09-27T16:48:41.8011989Z 
2022-09-27T16:48:41.8012117Z Generating XML reports...
2022-09-27T16:48:41.8046702Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164841.xml
2022-09-27T16:48:43.1128908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:43.1129425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:43.1130013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:43.1130476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:43.1131076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:43.1131560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:43.1132160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:43.1132647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:43.6418441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:43.6419051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:43.6419852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:43.6420563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:43.6426010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:43.6426912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:43.8069672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:43.8070327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:43.8072499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:43.8072975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:44.3252927Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:44.3276588Z 
2022-09-27T16:48:44.3277098Z Running tests...
2022-09-27T16:48:44.3277634Z ----------------------------------------------------------------------
2022-09-27T16:48:44.3296751Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-09-27T16:48:44.3297119Z 
2022-09-27T16:48:44.3297733Z ----------------------------------------------------------------------
2022-09-27T16:48:44.3298072Z Ran 1 test in 0.002s
2022-09-27T16:48:44.3298242Z 
2022-09-27T16:48:44.3298364Z OK (skipped=1)
2022-09-27T16:48:44.3298524Z 
2022-09-27T16:48:44.3298657Z Generating XML reports...
2022-09-27T16:48:44.3333759Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164844.xml
2022-09-27T16:48:44.9661197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm11jjvj1
2022-09-27T16:48:44.9662273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm11jjvj1/_remote_module_non_scriptable.py
2022-09-27T16:48:44.9818499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81kozx19
2022-09-27T16:48:44.9821042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81kozx19/_remote_module_non_scriptable.py
2022-09-27T16:48:46.0331606Z ok (6.209s)
2022-09-27T16:48:46.0331861Z 
2022-09-27T16:48:46.0332258Z ----------------------------------------------------------------------
2022-09-27T16:48:46.0332608Z Ran 1 test in 6.210s
2022-09-27T16:48:46.0333036Z 
2022-09-27T16:48:46.0333133Z OK
2022-09-27T16:48:46.0333270Z 
2022-09-27T16:48:46.0333413Z Generating XML reports...
2022-09-27T16:48:46.0367610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164839.xml
2022-09-27T16:48:46.3432914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:46.3433413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:46.3435644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:46.3436142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:46.8511412Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:46.8534559Z 
2022-09-27T16:48:46.8534710Z Running tests...
2022-09-27T16:48:46.8535448Z ----------------------------------------------------------------------
2022-09-27T16:48:46.8554464Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-09-27T16:48:46.8554827Z 
2022-09-27T16:48:46.8555120Z ----------------------------------------------------------------------
2022-09-27T16:48:46.8555457Z Ran 1 test in 0.002s
2022-09-27T16:48:46.8555605Z 
2022-09-27T16:48:46.8555719Z OK (skipped=1)
2022-09-27T16:48:46.8555881Z 
2022-09-27T16:48:46.8556011Z Generating XML reports...
2022-09-27T16:48:46.8590376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164846.xml
2022-09-27T16:48:48.0349568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:48.0350069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:48.0351327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:48.0351832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:48.5123084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:48.5145224Z 
2022-09-27T16:48:48.5145499Z Running tests...
2022-09-27T16:48:48.5145942Z ----------------------------------------------------------------------
2022-09-27T16:48:48.7669636Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:48.7670244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:48.7671867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:48.7672384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:49.2534050Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:49.2555689Z 
2022-09-27T16:48:49.2555808Z Running tests...
2022-09-27T16:48:49.2556886Z ----------------------------------------------------------------------
2022-09-27T16:48:49.2574613Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:48:49.2575037Z 
2022-09-27T16:48:49.2575567Z ----------------------------------------------------------------------
2022-09-27T16:48:49.2576312Z Ran 1 test in 0.002s
2022-09-27T16:48:49.2576614Z 
2022-09-27T16:48:49.2576740Z OK (skipped=1)
2022-09-27T16:48:49.2576901Z 
2022-09-27T16:48:49.2577689Z Generating XML reports...
2022-09-27T16:48:49.2609409Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164849.xml
2022-09-27T16:48:50.0288409Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:50.0479694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124244
2022-09-27T16:48:50.0486602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124245
2022-09-27T16:48:51.2754126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:51.2755107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:51.2756362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:51.2757325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:51.6955337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:51.6956331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:51.6957480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:51.6958475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:51.7652856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:51.7653724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:51.7655014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:51.7655848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:51.7662054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:48:51.7686537Z 
2022-09-27T16:48:51.7687015Z Running tests...
2022-09-27T16:48:51.7687755Z ----------------------------------------------------------------------
2022-09-27T16:48:52.2926196Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:52.2926881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:52.2927667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:52.2928371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:52.3034357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:52.3941912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:53.3332090Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:53.3536946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124354
2022-09-27T16:48:53.3543296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124355
2022-09-27T16:48:54.9407685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:54.9408210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:54.9409176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:54.9409653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:54.9910258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:54.9910743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:54.9912081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:54.9912551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:55.2597051Z ok (6.745s)
2022-09-27T16:48:55.2597470Z 
2022-09-27T16:48:55.2598221Z ----------------------------------------------------------------------
2022-09-27T16:48:55.2598620Z Ran 1 test in 6.745s
2022-09-27T16:48:55.2598772Z 
2022-09-27T16:48:55.2598869Z OK
2022-09-27T16:48:55.2599005Z 
2022-09-27T16:48:55.2599144Z Generating XML reports...
2022-09-27T16:48:55.2634891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164848.xml
2022-09-27T16:48:55.4738922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:48:55.5049789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:48:55.5051089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:55.5144598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:48:55.5151147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:48:55.6062584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:48:57.3583091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:57.3583650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:57.3585885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:57.5636914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:48:57.5637283Z ok (5.795s)
2022-09-27T16:48:57.5637441Z 
2022-09-27T16:48:57.5637825Z ----------------------------------------------------------------------
2022-09-27T16:48:57.5638163Z Ran 1 test in 5.795s
2022-09-27T16:48:57.5638345Z 
2022-09-27T16:48:57.5638438Z OK
2022-09-27T16:48:57.5638556Z 
2022-09-27T16:48:57.5638688Z Generating XML reports...
2022-09-27T16:48:57.5672569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164851.xml
2022-09-27T16:48:57.8714900Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:48:57.8739105Z 
2022-09-27T16:48:57.8739393Z Running tests...
2022-09-27T16:48:57.8739830Z ----------------------------------------------------------------------
2022-09-27T16:48:59.5198282Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:48:59.5396994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124507
2022-09-27T16:48:59.5403375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124508
2022-09-27T16:48:59.5583419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:48:59.5583928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:48:59.5587455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:48:59.5587968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:00.0388822Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:00.0410420Z 
2022-09-27T16:49:00.0410769Z Running tests...
2022-09-27T16:49:00.0411631Z ----------------------------------------------------------------------
2022-09-27T16:49:01.1668293Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:01.1668914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:01.1669821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:01.1670314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:01.1838162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:01.1838641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:01.1841904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:01.1842409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:01.6303889Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:01.6503821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124576
2022-09-27T16:49:01.6508554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124577
2022-09-27T16:49:01.7133270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:01.7137356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:01.7138947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:01.7234144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:01.7246900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:01.8249743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:03.2445970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:03.2446493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:03.2447721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:03.2448196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:03.2786194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:03.2786694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:03.2789384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:03.2789857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:03.7785372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:03.7991389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:03.7992227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:03.8089702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:03.8096454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:03.9007270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:04.7528372Z ok (6.878s)
2022-09-27T16:49:04.7528585Z 
2022-09-27T16:49:04.7528992Z ----------------------------------------------------------------------
2022-09-27T16:49:04.7529322Z Ran 1 test in 6.879s
2022-09-27T16:49:04.7529492Z 
2022-09-27T16:49:04.7529612Z OK
2022-09-27T16:49:04.7529750Z 
2022-09-27T16:49:04.7529884Z Generating XML reports...
2022-09-27T16:49:04.7567276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164857.xml
2022-09-27T16:49:05.5360031Z STAGE:2022-09-27 16:49:05 124576:124576 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:05.5734432Z STAGE:2022-09-27 16:49:05 124577:124577 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:05.6657828Z STAGE:2022-09-27 16:49:05 124577:124577 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:05.6658476Z STAGE:2022-09-27 16:49:05 124576:124576 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:06.4614901Z ok (6.420s)
2022-09-27T16:49:06.4615112Z 
2022-09-27T16:49:06.4615512Z ----------------------------------------------------------------------
2022-09-27T16:49:06.4615837Z Ran 1 test in 6.420s
2022-09-27T16:49:06.4616028Z 
2022-09-27T16:49:06.4616122Z OK
2022-09-27T16:49:06.4616260Z 
2022-09-27T16:49:06.4616406Z Generating XML reports...
2022-09-27T16:49:06.4651501Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164900.xml
2022-09-27T16:49:06.8818816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:06.8819313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:06.8822252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:06.8822753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:07.3821826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:49:07.3843704Z 
2022-09-27T16:49:07.3843966Z Running tests...
2022-09-27T16:49:07.3844428Z ----------------------------------------------------------------------
2022-09-27T16:49:08.4776592Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:08.4777239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:08.4778578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:08.4779051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:08.8993722Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:08.9184657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124744
2022-09-27T16:49:08.9191364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124745
2022-09-27T16:49:09.0001571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:09.0025622Z 
2022-09-27T16:49:09.0025807Z Running tests...
2022-09-27T16:49:09.0026480Z ----------------------------------------------------------------------
2022-09-27T16:49:10.5315865Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:10.5316476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:10.5317560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:10.5318063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:10.5914028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:10.5914530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:10.5917545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:10.5918046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:10.6032820Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:10.6233707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124812
2022-09-27T16:49:10.6239375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124813
2022-09-27T16:49:11.1258168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:11.1258733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:11.1259498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:11.1260222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:11.1266237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:11.1266738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:12.2709557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:12.2710077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:12.2711276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:12.2711750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:12.3181673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:12.3182176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:12.3184173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:12.3184640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:12.5076790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lw_noqb
2022-09-27T16:49:12.5077402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lw_noqb/_remote_module_non_scriptable.py
2022-09-27T16:49:12.5104896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxivypt6y
2022-09-27T16:49:12.5107872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxivypt6y/_remote_module_non_scriptable.py
2022-09-27T16:49:12.8162063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:12.8506056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:12.8507151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:12.8568491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:12.8575048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:12.9521233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:12.9642283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:12.9642856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:12.9667020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:12.9667549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:13.6295972Z ok (6.245s)
2022-09-27T16:49:13.6296230Z 
2022-09-27T16:49:13.6296603Z ----------------------------------------------------------------------
2022-09-27T16:49:13.6296949Z Ran 1 test in 6.245s
2022-09-27T16:49:13.6297407Z 
2022-09-27T16:49:13.6297504Z OK
2022-09-27T16:49:13.6297641Z 
2022-09-27T16:49:13.6298650Z Generating XML reports...
2022-09-27T16:49:13.6332550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164907.xml
2022-09-27T16:49:14.5865530Z STAGE:2022-09-27 16:49:14 124813:124813 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:14.6241486Z STAGE:2022-09-27 16:49:14 124812:124812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:14.7403262Z STAGE:2022-09-27 16:49:14 124813:124813 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:14.7403946Z STAGE:2022-09-27 16:49:14 124812:124812 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:15.5346316Z ok (6.532s)
2022-09-27T16:49:15.5346546Z 
2022-09-27T16:49:15.5346918Z ----------------------------------------------------------------------
2022-09-27T16:49:15.5347264Z Ran 1 test in 6.532s
2022-09-27T16:49:15.5347445Z 
2022-09-27T16:49:15.5349447Z OK
2022-09-27T16:49:15.5349770Z 
2022-09-27T16:49:15.5350136Z Generating XML reports...
2022-09-27T16:49:15.5464063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164908.xml
2022-09-27T16:49:15.7719665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:15.7720464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:15.7722048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:15.7722524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:16.2696590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:49:16.2717827Z 
2022-09-27T16:49:16.2718321Z Running tests...
2022-09-27T16:49:16.2719172Z ----------------------------------------------------------------------
2022-09-27T16:49:17.5763612Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:17.5764231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:17.5766349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:17.5766853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:17.8506614Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:17.8699239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124983
2022-09-27T16:49:17.8705799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124984
2022-09-27T16:49:18.0827378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:18.0850869Z 
2022-09-27T16:49:18.0850990Z Running tests...
2022-09-27T16:49:18.0851594Z ----------------------------------------------------------------------
2022-09-27T16:49:18.0870837Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:49:18.0871347Z 
2022-09-27T16:49:18.0871972Z ----------------------------------------------------------------------
2022-09-27T16:49:18.0872352Z Ran 1 test in 0.002s
2022-09-27T16:49:18.0872531Z 
2022-09-27T16:49:18.0872648Z OK (skipped=1)
2022-09-27T16:49:18.0872811Z 
2022-09-27T16:49:18.0872942Z Generating XML reports...
2022-09-27T16:49:18.0907294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164918.xml
2022-09-27T16:49:19.5248453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:19.5249292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:19.5250258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:19.5250734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:19.5540684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:19.5541156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:19.5545021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:19.5545497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:20.0411970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:20.0412484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:20.0414633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:20.0415108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:20.0760360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:20.0761106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:20.0761805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:20.0762498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:20.0870270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:20.1776912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:20.5406645Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:20.5428561Z 
2022-09-27T16:49:20.5429035Z Running tests...
2022-09-27T16:49:20.5429532Z ----------------------------------------------------------------------
2022-09-27T16:49:20.5446895Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:49:20.5447228Z 
2022-09-27T16:49:20.5447520Z ----------------------------------------------------------------------
2022-09-27T16:49:20.5447850Z Ran 1 test in 0.002s
2022-09-27T16:49:20.5448000Z 
2022-09-27T16:49:20.5448112Z OK (skipped=1)
2022-09-27T16:49:20.5448268Z 
2022-09-27T16:49:20.5448717Z Generating XML reports...
2022-09-27T16:49:20.5482714Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164920.xml
2022-09-27T16:49:21.4299614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz_t5l3j
2022-09-27T16:49:21.4300276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz_t5l3j/_remote_module_non_scriptable.py
2022-09-27T16:49:21.4787544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfwircai
2022-09-27T16:49:21.4788368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfwircai/_remote_module_non_scriptable.py
2022-09-27T16:49:21.9906590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:21.9907149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:21.9930148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:21.9930643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:22.5691617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:22.5692528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:22.5693823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:22.5694312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:22.5807875Z ok (6.309s)
2022-09-27T16:49:22.5808158Z 
2022-09-27T16:49:22.5808679Z ----------------------------------------------------------------------
2022-09-27T16:49:22.5809198Z Ran 1 test in 6.309s
2022-09-27T16:49:22.5809372Z 
2022-09-27T16:49:22.5809473Z OK
2022-09-27T16:49:22.5809607Z 
2022-09-27T16:49:22.5809737Z Generating XML reports...
2022-09-27T16:49:22.5845781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164916.xml
2022-09-27T16:49:23.0758416Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:23.0780303Z 
2022-09-27T16:49:23.0780593Z Running tests...
2022-09-27T16:49:23.0781037Z ----------------------------------------------------------------------
2022-09-27T16:49:23.0798501Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:49:23.0798853Z 
2022-09-27T16:49:23.0799133Z ----------------------------------------------------------------------
2022-09-27T16:49:23.0799442Z Ran 1 test in 0.002s
2022-09-27T16:49:23.0799605Z 
2022-09-27T16:49:23.0799717Z OK (skipped=1)
2022-09-27T16:49:23.0799878Z 
2022-09-27T16:49:23.0800003Z Generating XML reports...
2022-09-27T16:49:23.0834091Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164923.xml
2022-09-27T16:49:24.6641735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:24.6642309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:24.6645163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:24.6645653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:25.0080445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:25.0080956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:25.0083894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:25.0084667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:25.1541172Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:49:25.1564895Z 
2022-09-27T16:49:25.1565217Z Running tests...
2022-09-27T16:49:25.1565677Z ----------------------------------------------------------------------
2022-09-27T16:49:25.4942524Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:25.4964903Z 
2022-09-27T16:49:25.4965159Z Running tests...
2022-09-27T16:49:25.4965597Z ----------------------------------------------------------------------
2022-09-27T16:49:25.4984668Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-09-27T16:49:25.4985011Z 
2022-09-27T16:49:25.4985280Z ----------------------------------------------------------------------
2022-09-27T16:49:25.4985627Z Ran 1 test in 0.002s
2022-09-27T16:49:25.4985792Z 
2022-09-27T16:49:25.4985902Z OK (skipped=1)
2022-09-27T16:49:25.4986059Z 
2022-09-27T16:49:25.4986187Z Generating XML reports...
2022-09-27T16:49:25.5020948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164925.xml
2022-09-27T16:49:26.7354328Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:26.7553277Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125229
2022-09-27T16:49:26.7559985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125230
2022-09-27T16:49:27.3495071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:27.3495578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:27.3497879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:27.3498373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:27.8876936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:27.8899293Z 
2022-09-27T16:49:27.8899525Z Running tests...
2022-09-27T16:49:27.8899942Z ----------------------------------------------------------------------
2022-09-27T16:49:27.8916213Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-09-27T16:49:27.8917527Z 
2022-09-27T16:49:27.8917828Z ----------------------------------------------------------------------
2022-09-27T16:49:27.8918175Z Ran 1 test in 0.002s
2022-09-27T16:49:27.8918323Z 
2022-09-27T16:49:27.8918437Z OK (skipped=1)
2022-09-27T16:49:27.8918590Z 
2022-09-27T16:49:27.8918716Z Generating XML reports...
2022-09-27T16:49:27.8952299Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164927.xml
2022-09-27T16:49:28.3464114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:28.3464657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:28.3466688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:28.3467167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:28.3870930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:28.3871415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:28.3875067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:28.3875561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:29.0031822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:29.0032610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:29.0033583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:29.0034288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:29.0041241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:29.0042721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:29.3625989Z skip: Need at least 4 CUDA devices (4.206s)
2022-09-27T16:49:29.3626233Z 
2022-09-27T16:49:29.3626626Z ----------------------------------------------------------------------
2022-09-27T16:49:29.3626944Z Ran 1 test in 4.206s
2022-09-27T16:49:29.3627135Z 
2022-09-27T16:49:29.3627248Z OK (skipped=1)
2022-09-27T16:49:29.3627407Z 
2022-09-27T16:49:29.3627532Z Generating XML reports...
2022-09-27T16:49:29.3663346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164925.xml
2022-09-27T16:49:29.8424785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:29.8425281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:29.8426654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:29.8427144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:30.3322542Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:30.3345890Z 
2022-09-27T16:49:30.3346144Z Running tests...
2022-09-27T16:49:30.3346595Z ----------------------------------------------------------------------
2022-09-27T16:49:30.3365044Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-09-27T16:49:30.3365396Z 
2022-09-27T16:49:30.3365676Z ----------------------------------------------------------------------
2022-09-27T16:49:30.3366009Z Ran 1 test in 0.002s
2022-09-27T16:49:30.3366158Z 
2022-09-27T16:49:30.3366269Z OK (skipped=1)
2022-09-27T16:49:30.3366430Z 
2022-09-27T16:49:30.3366556Z Generating XML reports...
2022-09-27T16:49:30.3401130Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164930.xml
2022-09-27T16:49:31.3968832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:31.3970009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:31.3972796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:31.3973936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:31.8881802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:49:31.8905942Z 
2022-09-27T16:49:31.8906231Z Running tests...
2022-09-27T16:49:31.8906693Z ----------------------------------------------------------------------
2022-09-27T16:49:32.2366212Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:32.2366882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:32.2369605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:32.2370443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:32.7240788Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:32.7263785Z 
2022-09-27T16:49:32.7264023Z Running tests...
2022-09-27T16:49:32.7264469Z ----------------------------------------------------------------------
2022-09-27T16:49:33.4920047Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:33.5124409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125406
2022-09-27T16:49:33.5130925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125407
2022-09-27T16:49:34.2636161Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:34.2830193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125471
2022-09-27T16:49:34.2836512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125472
2022-09-27T16:49:35.1774151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:35.1774675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:35.1775626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:35.1776120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:35.1966472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:35.1966950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:35.1969915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:35.1970402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:35.7344884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:35.7345438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:35.7346236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:35.7346944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:35.7353808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:35.7354782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:35.9208554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:35.9209062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:35.9210662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:35.9211136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:35.9402337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:35.9402827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:35.9405534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:35.9406001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:36.1193964Z skip: Need at least 4 CUDA devices (4.228s)
2022-09-27T16:49:36.1194193Z 
2022-09-27T16:49:36.1194571Z ----------------------------------------------------------------------
2022-09-27T16:49:36.1194908Z Ran 1 test in 4.229s
2022-09-27T16:49:36.1195055Z 
2022-09-27T16:49:36.1195168Z OK (skipped=1)
2022-09-27T16:49:36.1195588Z 
2022-09-27T16:49:36.1195741Z Generating XML reports...
2022-09-27T16:49:36.1230629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164931.xml
2022-09-27T16:49:36.4370447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:36.4433287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:36.4434097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:36.4471824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:36.4478162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:36.5448030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:38.1635059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:38.1636413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:38.1637604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:38.1638570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:38.6888927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:49:38.6912082Z 
2022-09-27T16:49:38.6912580Z Running tests...
2022-09-27T16:49:38.6913100Z ----------------------------------------------------------------------
2022-09-27T16:49:39.8979639Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... ok (7.171s)
2022-09-27T16:49:39.8979980Z 
2022-09-27T16:49:39.8980398Z ----------------------------------------------------------------------
2022-09-27T16:49:39.8980746Z Ran 1 test in 7.172s
2022-09-27T16:49:39.8980918Z 
2022-09-27T16:49:39.8980996Z OK
2022-09-27T16:49:39.8981133Z 
2022-09-27T16:49:39.8981276Z Generating XML reports...
2022-09-27T16:49:39.9034537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164932.xml
2022-09-27T16:49:40.4243957Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:40.4444676Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125603
2022-09-27T16:49:40.4451641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125604
2022-09-27T16:49:42.1078829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:42.1079345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:42.1081452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:42.1081933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:42.1520911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:42.1521613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:42.1525040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:42.1525511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:42.1597234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:42.1597707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:42.1601498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:42.1602258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:42.6340843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:42.6364502Z 
2022-09-27T16:49:42.6364784Z Running tests...
2022-09-27T16:49:42.6365228Z ----------------------------------------------------------------------
2022-09-27T16:49:42.6386967Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-09-27T16:49:42.6387288Z 
2022-09-27T16:49:42.6387663Z ----------------------------------------------------------------------
2022-09-27T16:49:42.6388148Z Ran 1 test in 0.002s
2022-09-27T16:49:42.6388330Z 
2022-09-27T16:49:42.6388440Z OK (skipped=1)
2022-09-27T16:49:42.6388597Z 
2022-09-27T16:49:42.6388726Z Generating XML reports...
2022-09-27T16:49:42.6424687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164942.xml
2022-09-27T16:49:42.6963094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:42.6963637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:42.6964708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:42.6965413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:42.6972195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:42.6972689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:44.0441380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2jhh74b
2022-09-27T16:49:44.0442255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2jhh74b/_remote_module_non_scriptable.py
2022-09-27T16:49:44.0805578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vso2z8y
2022-09-27T16:49:44.0807053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vso2z8y/_remote_module_non_scriptable.py
2022-09-27T16:49:44.5109295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:44.5109854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-09-27T16:49:44.6194900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:44.6195406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:44.6196813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:44.6197334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:45.0553164Z ok (6.364s)
2022-09-27T16:49:45.0553510Z 
2022-09-27T16:49:45.0554160Z ----------------------------------------------------------------------
2022-09-27T16:49:45.0554833Z Ran 1 test in 6.364s
2022-09-27T16:49:45.0555145Z 
2022-09-27T16:49:45.0555318Z OK
2022-09-27T16:49:45.0555576Z 
2022-09-27T16:49:45.0555813Z Generating XML reports...
2022-09-27T16:49:45.0592704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164938.xml
2022-09-27T16:49:45.1482000Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:45.1503511Z 
2022-09-27T16:49:45.1503930Z Running tests...
2022-09-27T16:49:45.1504435Z ----------------------------------------------------------------------
2022-09-27T16:49:46.7503874Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:46.7696864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125782
2022-09-27T16:49:46.7702795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125783
2022-09-27T16:49:47.0885800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:47.0886311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:47.0887495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:47.0887982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:47.6274815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:49:47.6297247Z 
2022-09-27T16:49:47.6297546Z Running tests...
2022-09-27T16:49:47.6298021Z ----------------------------------------------------------------------
2022-09-27T16:49:48.3717219Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:48.3717850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:48.3719442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:48.3719912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:48.3827972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:48.3828443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:48.3832048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:48.3832514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:48.8873030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:48.8999617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:48.9000643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:48.9076149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:48.9083361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:49.0014685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:49.1740986Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:49.1941215Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125859
2022-09-27T16:49:49.1946272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125860
2022-09-27T16:49:50.2046561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyn6587cg
2022-09-27T16:49:50.2047712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyn6587cg/_remote_module_non_scriptable.py
2022-09-27T16:49:50.3755438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdmxh6f8
2022-09-27T16:49:50.3756384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdmxh6f8/_remote_module_non_scriptable.py
2022-09-27T16:49:50.8271245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:50.8271757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:50.8272802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:50.8273293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:50.8280934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:50.8281656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:50.8285737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:50.8286227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:51.3764343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:51.3764889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:51.3765706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:51.3766419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:51.3773410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:51.3774420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:51.3984613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:49:51.3985188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:49:51.3985988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:49:51.3986851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:49:51.4051558Z STAGE:2022-09-27 16:49:51 125859:125859 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:51.4147401Z STAGE:2022-09-27 16:49:51 125860:125860 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:51.4160119Z STAGE:2022-09-27 16:49:51 125860:125860 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:51.4161004Z STAGE:2022-09-27 16:49:51 125859:125859 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:51.4177840Z STAGE:2022-09-27 16:49:51 125860:125860 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:51.4185564Z STAGE:2022-09-27 16:49:51 125859:125859 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:51.4199658Z STAGE:2022-09-27 16:49:51 125859:125859 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:51.4200503Z STAGE:2022-09-27 16:49:51 125860:125860 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:51.8827882Z ok (6.732s)
2022-09-27T16:49:51.8828061Z 
2022-09-27T16:49:51.8828441Z ----------------------------------------------------------------------
2022-09-27T16:49:51.8828801Z Ran 1 test in 6.732s
2022-09-27T16:49:51.8828955Z 
2022-09-27T16:49:51.8829050Z OK
2022-09-27T16:49:51.8829188Z 
2022-09-27T16:49:51.8829319Z Generating XML reports...
2022-09-27T16:49:51.8864805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164945.xml
2022-09-27T16:49:51.9015787Z ok (4.272s)
2022-09-27T16:49:51.9015972Z 
2022-09-27T16:49:51.9016270Z ----------------------------------------------------------------------
2022-09-27T16:49:51.9016617Z Ran 1 test in 4.272s
2022-09-27T16:49:51.9016784Z 
2022-09-27T16:49:51.9016883Z OK
2022-09-27T16:49:51.9017023Z 
2022-09-27T16:49:51.9017134Z Generating XML reports...
2022-09-27T16:49:51.9053349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164947.xml
2022-09-27T16:49:54.0188730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:54.0189687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:54.0192881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:54.0193603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:54.0433706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:54.0434198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:54.0437515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:54.0438014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:54.5223178Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:49:54.5245805Z 
2022-09-27T16:49:54.5246052Z Running tests...
2022-09-27T16:49:54.5246485Z ----------------------------------------------------------------------
2022-09-27T16:49:54.5422311Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:49:54.5446591Z 
2022-09-27T16:49:54.5446933Z Running tests...
2022-09-27T16:49:54.5447373Z ----------------------------------------------------------------------
2022-09-27T16:49:56.0894772Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:56.1089530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126020
2022-09-27T16:49:56.1095305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126021
2022-09-27T16:49:56.1615181Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:49:56.1819552Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126023
2022-09-27T16:49:56.1825228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126024
2022-09-27T16:49:57.7952377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:57.7952940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:57.7953968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:57.7954591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:57.8299379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:57.8300023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:57.8303288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:57.8303860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:57.8665333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:57.8666273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:57.8669283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:57.8670232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:57.9023790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:49:57.9024687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:49:57.9025343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:49:57.9025822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:49:58.3491169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:58.3491957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:58.3492748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:58.3493448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:58.3499298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:58.3500039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:58.3591863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:49:58.3606797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:49:58.3607912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:49:58.3608762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:49:58.3609760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:49:58.3702755Z STAGE:2022-09-27 16:49:58 126020:126020 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:58.3703849Z STAGE:2022-09-27 16:49:58 126021:126021 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:58.3715966Z STAGE:2022-09-27 16:49:58 126020:126020 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:58.3717048Z STAGE:2022-09-27 16:49:58 126021:126021 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:58.3732805Z STAGE:2022-09-27 16:49:58 126021:126021 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:58.3740450Z STAGE:2022-09-27 16:49:58 126020:126020 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:49:58.3753801Z STAGE:2022-09-27 16:49:58 126020:126020 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:58.3754423Z STAGE:2022-09-27 16:49:58 126021:126021 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:49:58.3933823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:49:58.3934559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:58.3998982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:49:58.4005809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:49:58.4948907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:49:58.8160790Z ok (4.291s)
2022-09-27T16:49:58.8161287Z 
2022-09-27T16:49:58.8161674Z ----------------------------------------------------------------------
2022-09-27T16:49:58.8162021Z Ran 1 test in 4.291s
2022-09-27T16:49:58.8162193Z 
2022-09-27T16:49:58.8162270Z OK
2022-09-27T16:49:58.8162410Z 
2022-09-27T16:49:58.8164090Z Generating XML reports...
2022-09-27T16:49:58.8196910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164954.xml
2022-09-27T16:49:59.7513762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmd71yra
2022-09-27T16:49:59.7514408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmd71yra/_remote_module_non_scriptable.py
2022-09-27T16:49:59.8082905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbq_5s6xc
2022-09-27T16:49:59.8084407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbq_5s6xc/_remote_module_non_scriptable.py
2022-09-27T16:50:01.0018665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:01.0019240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:01.0020819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:01.0021311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:01.3582300Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:50:01.3586757Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:50:01.5388449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:01.5412532Z 
2022-09-27T16:50:01.5412917Z Running tests...
2022-09-27T16:50:01.5413418Z ----------------------------------------------------------------------
2022-09-27T16:50:01.9952950Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... ok (7.450s)
2022-09-27T16:50:01.9953297Z 
2022-09-27T16:50:01.9953958Z ----------------------------------------------------------------------
2022-09-27T16:50:01.9954340Z Ran 1 test in 7.450s
2022-09-27T16:50:01.9954507Z 
2022-09-27T16:50:01.9954601Z OK
2022-09-27T16:50:01.9954742Z 
2022-09-27T16:50:01.9954853Z Generating XML reports...
2022-09-27T16:50:01.9991110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164954.xml
2022-09-27T16:50:03.2664954Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:03.2860859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126258
2022-09-27T16:50:03.2869012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126259
2022-09-27T16:50:04.1495014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:04.1495545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:04.1497052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:04.1497913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:04.6349662Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:50:04.6371263Z 
2022-09-27T16:50:04.6371637Z Running tests...
2022-09-27T16:50:04.6372528Z ----------------------------------------------------------------------
2022-09-27T16:50:04.9197219Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:04.9197941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:04.9199589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:04.9200129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:04.9273462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:04.9274278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:04.9277453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:04.9278264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:05.4634026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:05.4635028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:05.4635996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:05.4636711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:05.4642601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:05.4643700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:05.4851496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:50:05.4852643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:50:05.4853709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:50:05.4854411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:50:05.4946875Z STAGE:2022-09-27 16:50:05 126258:126258 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:05.4948010Z STAGE:2022-09-27 16:50:05 126259:126259 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:05.4960211Z STAGE:2022-09-27 16:50:05 126259:126259 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:05.4961044Z STAGE:2022-09-27 16:50:05 126258:126258 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:05.4978035Z STAGE:2022-09-27 16:50:05 126259:126259 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:05.4984026Z STAGE:2022-09-27 16:50:05 126258:126258 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:05.4997478Z STAGE:2022-09-27 16:50:05 126258:126258 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:05.4998088Z STAGE:2022-09-27 16:50:05 126259:126259 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:05.8932580Z ok (4.352s)
2022-09-27T16:50:05.8932796Z 
2022-09-27T16:50:05.8933169Z ----------------------------------------------------------------------
2022-09-27T16:50:05.8933512Z Ran 1 test in 4.352s
2022-09-27T16:50:05.8933681Z 
2022-09-27T16:50:05.8933777Z OK
2022-09-27T16:50:05.8933919Z 
2022-09-27T16:50:05.8934055Z Generating XML reports...
2022-09-27T16:50:05.8969653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165001.xml
2022-09-27T16:50:06.1865894Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:06.2064281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126345
2022-09-27T16:50:06.2070075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126346
2022-09-27T16:50:07.8817061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:07.8818018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:07.8819387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:07.8819980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:07.9349566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:07.9350040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:07.9353050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:07.9353560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:07.9981483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:07.9981969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:07.9984567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:07.9985081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:08.3801678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:08.4417650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:08.4418715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:08.4510728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:08.4517676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:08.4987236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:08.5011840Z 
2022-09-27T16:50:08.5012223Z Running tests...
2022-09-27T16:50:08.5012737Z ----------------------------------------------------------------------
2022-09-27T16:50:08.5428482Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:08.5636122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:50:08.5636761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:50:08.5637497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:50:08.5638205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:50:08.5638951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:50:08.5640871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:50:08.5641565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:50:08.5742385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:50:09.9523302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1kteoqv
2022-09-27T16:50:09.9524091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1kteoqv/_remote_module_non_scriptable.py
2022-09-27T16:50:09.9814729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjs0wzhc
2022-09-27T16:50:09.9816104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjs0wzhc/_remote_module_non_scriptable.py
2022-09-27T16:50:10.1993077Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:10.2191273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126469
2022-09-27T16:50:10.2197779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126470
2022-09-27T16:50:11.8042478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:11.8043128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:11.8044154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:11.8044627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:11.8289163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:11.8289642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:11.8293070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:11.8293551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:12.3442928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:12.3443496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:12.3444286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:12.3444975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:12.3451294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:12.3451788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:12.3558756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:50:12.3559259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:50:12.3559965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:50:12.3561075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:50:12.3623053Z STAGE:2022-09-27 16:50:12 126469:126469 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:12.3720479Z STAGE:2022-09-27 16:50:12 126470:126470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:12.3733413Z STAGE:2022-09-27 16:50:12 126470:126470 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:12.3734014Z STAGE:2022-09-27 16:50:12 126469:126469 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:12.3748769Z STAGE:2022-09-27 16:50:12 126470:126470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:12.3755664Z STAGE:2022-09-27 16:50:12 126469:126469 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:12.3769596Z STAGE:2022-09-27 16:50:12 126469:126469 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:12.3770190Z STAGE:2022-09-27 16:50:12 126470:126470 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:12.7260198Z ok (4.224s)
2022-09-27T16:50:12.7260414Z 
2022-09-27T16:50:12.7260803Z ----------------------------------------------------------------------
2022-09-27T16:50:12.7261164Z Ran 1 test in 4.225s
2022-09-27T16:50:12.7261317Z 
2022-09-27T16:50:12.7261418Z OK
2022-09-27T16:50:12.7261556Z 
2022-09-27T16:50:12.7261691Z Generating XML reports...
2022-09-27T16:50:12.7298105Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165008.xml
2022-09-27T16:50:14.7072660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:14.7073197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:14.7075536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:14.7076035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:15.2111902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:15.2135717Z 
2022-09-27T16:50:15.2136120Z Running tests...
2022-09-27T16:50:15.2136626Z ----------------------------------------------------------------------
2022-09-27T16:50:15.9282452Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... ok (11.291s)
2022-09-27T16:50:15.9282787Z 
2022-09-27T16:50:15.9283195Z ----------------------------------------------------------------------
2022-09-27T16:50:15.9283532Z Ran 1 test in 11.291s
2022-09-27T16:50:15.9283695Z 
2022-09-27T16:50:15.9283797Z OK
2022-09-27T16:50:15.9284236Z 
2022-09-27T16:50:15.9284349Z Generating XML reports...
2022-09-27T16:50:15.9344620Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165004.xml
2022-09-27T16:50:16.9236507Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:16.9429828Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126589
2022-09-27T16:50:16.9437385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126590
2022-09-27T16:50:18.1493481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:18.1494018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:18.1495470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:18.1495963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:18.6331104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-09-27T16:50:18.6353484Z 
2022-09-27T16:50:18.6353753Z Running tests...
2022-09-27T16:50:18.6354199Z ----------------------------------------------------------------------
2022-09-27T16:50:18.6574519Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:18.6575102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:18.6576526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:18.6577042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:18.6720752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:18.6721264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:18.6724651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:18.6725139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:19.2119963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:19.2120958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:19.2121748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:19.2122655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:19.2128429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:19.2128939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:19.6503792Z skip: Skipped due to small world size. (4.436s)
2022-09-27T16:50:19.6504040Z 
2022-09-27T16:50:19.6504435Z ----------------------------------------------------------------------
2022-09-27T16:50:19.6504776Z Ran 1 test in 4.437s
2022-09-27T16:50:19.6504945Z 
2022-09-27T16:50:19.6505045Z OK (skipped=1)
2022-09-27T16:50:19.6505202Z 
2022-09-27T16:50:19.6505331Z Generating XML reports...
2022-09-27T16:50:19.6540205Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165015.xml
2022-09-27T16:50:20.1658730Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:20.1853189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126698
2022-09-27T16:50:20.1859524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126699
2022-09-27T16:50:21.7591566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:21.7592583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:21.7593652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:21.7594147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:21.8648689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:21.8649602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:21.8650847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:21.8651343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:21.8918718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:21.8919203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:21.8923169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:21.8923658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:22.2432171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:22.2455194Z 
2022-09-27T16:50:22.2455647Z Running tests...
2022-09-27T16:50:22.2456120Z ----------------------------------------------------------------------
2022-09-27T16:50:22.3570778Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:22.3982236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:22.3983080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:22.4078027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:22.4084749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:22.4995467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:22.5104300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:50:22.5105345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:50:22.5106455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:50:22.5107190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:50:22.5110448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:50:22.5209990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:50:22.5211537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:50:22.5213791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:50:23.8964126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9wmomdh
2022-09-27T16:50:23.8965895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9wmomdh/_remote_module_non_scriptable.py
2022-09-27T16:50:23.9058095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7g961qo5
2022-09-27T16:50:23.9060293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7g961qo5/_remote_module_non_scriptable.py
2022-09-27T16:50:23.9194355Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:23.9388963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126819
2022-09-27T16:50:23.9395337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126820
2022-09-27T16:50:25.5598612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:25.5599134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:25.5600633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:25.5601161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:25.5895013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:25.5895485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:25.5899206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:25.5899715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:26.1094829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:26.1095994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:26.1096820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:26.1097541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:26.1102803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:26.1104092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:26.4453785Z skip: Skipped due to small world size. (4.199s)
2022-09-27T16:50:26.4454228Z 
2022-09-27T16:50:26.4454687Z ----------------------------------------------------------------------
2022-09-27T16:50:26.4455038Z Ran 1 test in 4.200s
2022-09-27T16:50:26.4455188Z 
2022-09-27T16:50:26.4455300Z OK (skipped=1)
2022-09-27T16:50:26.4455580Z 
2022-09-27T16:50:26.4455802Z Generating XML reports...
2022-09-27T16:50:26.4492087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165022.xml
2022-09-27T16:50:28.4233939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:28.4234489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:28.4236175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:28.4236681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:28.9208549Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:28.9231060Z 
2022-09-27T16:50:28.9231392Z Running tests...
2022-09-27T16:50:28.9231844Z ----------------------------------------------------------------------
2022-09-27T16:50:30.1050026Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... ok (11.469s)
2022-09-27T16:50:30.1050346Z 
2022-09-27T16:50:30.1050742Z ----------------------------------------------------------------------
2022-09-27T16:50:30.1051084Z Ran 1 test in 11.469s
2022-09-27T16:50:30.1051253Z 
2022-09-27T16:50:30.1051348Z OK
2022-09-27T16:50:30.1051487Z 
2022-09-27T16:50:30.1051618Z Generating XML reports...
2022-09-27T16:50:30.1086645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165018.xml
2022-09-27T16:50:31.0836165Z INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:31.1027190Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126930
2022-09-27T16:50:31.1033724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126931
2022-09-27T16:50:32.7050555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:32.7051104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:32.7052039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:32.7052539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:32.7130203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:32.7130678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:32.7134295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:32.7134795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:33.2354475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:33.2355049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:33.2355854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:33.2356547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:33.2362470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:33.2363452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:33.6102865Z skip: Skipped due to small world size. (4.687s)
2022-09-27T16:50:33.6103280Z 
2022-09-27T16:50:33.6103749Z ----------------------------------------------------------------------
2022-09-27T16:50:33.6104081Z Ran 1 test in 4.687s
2022-09-27T16:50:33.6104251Z 
2022-09-27T16:50:33.6104382Z OK (skipped=1)
2022-09-27T16:50:33.6104674Z 
2022-09-27T16:50:33.6104907Z Generating XML reports...
2022-09-27T16:50:33.6141029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165028.xml
2022-09-27T16:50:35.5936986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:35.5937857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:35.5939001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:35.5939669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:36.0891790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:36.0913600Z 
2022-09-27T16:50:36.0913931Z Running tests...
2022-09-27T16:50:36.0914369Z ----------------------------------------------------------------------
2022-09-27T16:50:37.6129033Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:37.6319325Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127039
2022-09-27T16:50:37.6326345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127040
2022-09-27T16:50:39.2763141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:39.2763659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:39.2766055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:39.2766881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:39.2849860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:39.2850337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:39.2853906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:39.2854404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:39.8144226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:39.8144782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:39.8145597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:39.8146318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:39.8152905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:39.8153533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:40.1387495Z skip: Skipped due to small world size. (4.047s)
2022-09-27T16:50:40.1387929Z 
2022-09-27T16:50:40.1388331Z ----------------------------------------------------------------------
2022-09-27T16:50:40.1388657Z Ran 1 test in 4.047s
2022-09-27T16:50:40.1388822Z 
2022-09-27T16:50:40.1388933Z OK (skipped=1)
2022-09-27T16:50:40.1389092Z 
2022-09-27T16:50:40.1389241Z Generating XML reports...
2022-09-27T16:50:40.1425513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165036.xml
2022-09-27T16:50:42.1138824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:42.1139354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:42.1140908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:42.1141382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:42.6179530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:42.6202728Z 
2022-09-27T16:50:42.6203131Z Running tests...
2022-09-27T16:50:42.6213933Z ----------------------------------------------------------------------
2022-09-27T16:50:44.1795560Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:44.1992117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127148
2022-09-27T16:50:44.1997930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127149
2022-09-27T16:50:45.7215644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:45.7216180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:45.7217373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:45.7217901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:45.7933073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:45.7933585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:45.7935894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:45.7936719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:46.3218848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:46.3219409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:46.3220206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:46.3220913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:46.3227290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:46.3227804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:46.3322449Z STAGE:2022-09-27 16:50:46 127149:127149 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:46.3323073Z STAGE:2022-09-27 16:50:46 127148:127148 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:46.3336969Z STAGE:2022-09-27 16:50:46 127149:127149 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:50:46 127148:127148 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:46.3337381Z 
2022-09-27T16:50:46.3352890Z STAGE:2022-09-27 16:50:46 127149:127149 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:46.3360090Z STAGE:2022-09-27 16:50:46 127148:127148 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:46.3374139Z STAGE:2022-09-27 16:50:46 127148:127148 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:50:46 127149:127149 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:46.3374549Z 
2022-09-27T16:50:46.7064404Z ok (4.086s)
2022-09-27T16:50:46.7064616Z 
2022-09-27T16:50:46.7065019Z ----------------------------------------------------------------------
2022-09-27T16:50:46.7065365Z Ran 1 test in 4.086s
2022-09-27T16:50:46.7065534Z 
2022-09-27T16:50:46.7065630Z OK
2022-09-27T16:50:46.7065765Z 
2022-09-27T16:50:46.7065899Z Generating XML reports...
2022-09-27T16:50:46.7101616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165042.xml
2022-09-27T16:50:48.6759240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:48.6759787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:48.6762798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:48.6763581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:49.1729434Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:49.1750854Z 
2022-09-27T16:50:49.1751014Z Running tests...
2022-09-27T16:50:49.1752018Z ----------------------------------------------------------------------
2022-09-27T16:50:50.6936849Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:50.7127301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127261
2022-09-27T16:50:50.7134000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127262
2022-09-27T16:50:52.3057094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:52.3057595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:52.3059045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:52.3059543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:52.3261668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:52.3262149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:52.3265464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:52.3265955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:52.8618020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:50:52.8618590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:50:52.8619390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:52.8620097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:50:52.8727152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:50:52.8791606Z STAGE:2022-09-27 16:50:52 127262:127262 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:52.9628987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:50:52.9692404Z STAGE:2022-09-27 16:50:52 127261:127261 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:52.9706348Z STAGE:2022-09-27 16:50:52 127262:127262 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:50:52 127261:127261 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:52.9706763Z 
2022-09-27T16:50:52.9722213Z STAGE:2022-09-27 16:50:52 127262:127262 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:52.9729953Z STAGE:2022-09-27 16:50:52 127261:127261 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:50:52.9743715Z STAGE:2022-09-27 16:50:52 127261:127261 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:52.9744326Z STAGE:2022-09-27 16:50:52 127262:127262 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:50:53.3195035Z ok (4.144s)
2022-09-27T16:50:53.3195241Z 
2022-09-27T16:50:53.3195638Z ----------------------------------------------------------------------
2022-09-27T16:50:53.3195981Z Ran 1 test in 4.144s
2022-09-27T16:50:53.3196150Z 
2022-09-27T16:50:53.3196248Z OK
2022-09-27T16:50:53.3196385Z 
2022-09-27T16:50:53.3196503Z Generating XML reports...
2022-09-27T16:50:53.3232121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165049.xml
2022-09-27T16:50:55.2934327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:55.2934886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:55.2936020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:55.2936552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:55.7965203Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:55.7988341Z 
2022-09-27T16:50:55.7988572Z Running tests...
2022-09-27T16:50:55.7989438Z ----------------------------------------------------------------------
2022-09-27T16:50:55.8010649Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-09-27T16:50:55.8011214Z 
2022-09-27T16:50:55.8011912Z ----------------------------------------------------------------------
2022-09-27T16:50:55.8012726Z Ran 1 test in 0.002s
2022-09-27T16:50:55.8013051Z 
2022-09-27T16:50:55.8013204Z OK (skipped=1)
2022-09-27T16:50:55.8013365Z 
2022-09-27T16:50:55.8013494Z Generating XML reports...
2022-09-27T16:50:55.8047127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165055.xml
2022-09-27T16:50:57.6455923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:50:57.6456442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:50:57.6457905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:50:57.6458390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:50:58.1506127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:50:58.1529432Z 
2022-09-27T16:50:58.1529660Z Running tests...
2022-09-27T16:50:58.1530103Z ----------------------------------------------------------------------
2022-09-27T16:50:59.6995881Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:50:59.7191748Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127407
2022-09-27T16:50:59.7198399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127408
2022-09-27T16:51:01.3316507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:01.3317006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:01.3319493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:01.3319992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:01.3643857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:01.3644319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:01.3647797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:01.3648582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:01.8937987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:51:01.8938716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:51:01.8939518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:01.8940227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:01.8947310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:51:01.8947992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:51:01.9045226Z STAGE:2022-09-27 16:51:01 127408:127408 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:01.9046032Z STAGE:2022-09-27 16:51:01 127407:127407 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:01.9059863Z STAGE:2022-09-27 16:51:01 127408:127408 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:51:01 127407:127407 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:01.9060404Z 
2022-09-27T16:51:01.9075917Z STAGE:2022-09-27 16:51:01 127408:127408 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:01.9084069Z STAGE:2022-09-27 16:51:01 127407:127407 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:01.9098184Z STAGE:2022-09-27 16:51:01 127407:127407 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:01.9099117Z STAGE:2022-09-27 16:51:01 127408:127408 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:02.3259191Z ok (4.173s)
2022-09-27T16:51:02.3259485Z 
2022-09-27T16:51:02.3259880Z ----------------------------------------------------------------------
2022-09-27T16:51:02.3260229Z Ran 1 test in 4.173s
2022-09-27T16:51:02.3260397Z 
2022-09-27T16:51:02.3260492Z OK
2022-09-27T16:51:02.3260627Z 
2022-09-27T16:51:02.3260750Z Generating XML reports...
2022-09-27T16:51:02.3295746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165058.xml
2022-09-27T16:51:04.3684517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:04.3685026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:04.3686600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:04.3687169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:04.8788085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:04.8811560Z 
2022-09-27T16:51:04.8812013Z Running tests...
2022-09-27T16:51:04.8812523Z ----------------------------------------------------------------------
2022-09-27T16:51:04.8840977Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-09-27T16:51:04.8841302Z 
2022-09-27T16:51:04.8841591Z ----------------------------------------------------------------------
2022-09-27T16:51:04.8841943Z Ran 1 test in 0.003s
2022-09-27T16:51:04.8842110Z 
2022-09-27T16:51:04.8842220Z OK (skipped=1)
2022-09-27T16:51:04.8842367Z 
2022-09-27T16:51:04.8842496Z Generating XML reports...
2022-09-27T16:51:04.8876483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165104.xml
2022-09-27T16:51:06.6953306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:06.6954390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:06.6955570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:06.6956539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:07.1997170Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:07.2022138Z 
2022-09-27T16:51:07.2022672Z Running tests...
2022-09-27T16:51:07.2023182Z ----------------------------------------------------------------------
2022-09-27T16:51:08.7622538Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:51:08.7819363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127553
2022-09-27T16:51:08.7825775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127554
2022-09-27T16:51:10.3645878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:10.3646919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:10.3648131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:10.3649081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:10.3878096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:10.3879041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:10.3882983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:10.3883986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:10.9181641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:51:10.9182702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:51:10.9184137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:10.9185517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:10.9190540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:51:10.9191468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:51:10.9287741Z STAGE:2022-09-27 16:51:10 127554:127554 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:10.9288359Z STAGE:2022-09-27 16:51:10 127553:127553 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:10.9302319Z STAGE:2022-09-27 16:51:10 127554:127554 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:10.9303527Z STAGE:2022-09-27 16:51:10 127553:127553 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:10.9319281Z STAGE:2022-09-27 16:51:10 127554:127554 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:10.9328457Z STAGE:2022-09-27 16:51:10 127553:127553 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:10.9343368Z STAGE:2022-09-27 16:51:10 127553:127553 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:51:10 127554:127554 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:10.9344208Z 
2022-09-27T16:51:11.2888131Z ok (4.086s)
2022-09-27T16:51:11.2888376Z 
2022-09-27T16:51:11.2888774Z ----------------------------------------------------------------------
2022-09-27T16:51:11.2889133Z Ran 1 test in 4.087s
2022-09-27T16:51:11.2889303Z 
2022-09-27T16:51:11.2889399Z OK
2022-09-27T16:51:11.2889534Z 
2022-09-27T16:51:11.2889673Z Generating XML reports...
2022-09-27T16:51:11.2926142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165107.xml
2022-09-27T16:51:13.2802697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:13.2803547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:13.2804781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:13.2805279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:13.7718902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:13.7741959Z 
2022-09-27T16:51:13.7742213Z Running tests...
2022-09-27T16:51:13.7742676Z ----------------------------------------------------------------------
2022-09-27T16:51:13.7764031Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:51:13.7764576Z 
2022-09-27T16:51:13.7765038Z ----------------------------------------------------------------------
2022-09-27T16:51:13.7765360Z Ran 1 test in 0.002s
2022-09-27T16:51:13.7765533Z 
2022-09-27T16:51:13.7765644Z OK (skipped=1)
2022-09-27T16:51:13.7765800Z 
2022-09-27T16:51:13.7765927Z Generating XML reports...
2022-09-27T16:51:13.7799165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165113.xml
2022-09-27T16:51:15.5785943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:15.5786442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:15.5788011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:15.5788811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:16.0737739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:16.0761052Z 
2022-09-27T16:51:16.0761514Z Running tests...
2022-09-27T16:51:16.0762022Z ----------------------------------------------------------------------
2022-09-27T16:51:16.0783219Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-09-27T16:51:16.0783549Z 
2022-09-27T16:51:16.0783843Z ----------------------------------------------------------------------
2022-09-27T16:51:16.0784160Z Ran 1 test in 0.002s
2022-09-27T16:51:16.0784342Z 
2022-09-27T16:51:16.0784454Z OK (skipped=1)
2022-09-27T16:51:16.0784613Z 
2022-09-27T16:51:16.0787128Z Generating XML reports...
2022-09-27T16:51:16.0819051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165116.xml
2022-09-27T16:51:17.9219588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:17.9220083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:17.9221061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:17.9221553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:18.4180986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:18.4203651Z 
2022-09-27T16:51:18.4203793Z Running tests...
2022-09-27T16:51:18.4204629Z ----------------------------------------------------------------------
2022-09-27T16:51:19.9577438Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:51:19.9766983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127732
2022-09-27T16:51:19.9773568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127733
2022-09-27T16:51:21.5652264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:21.5652785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:21.5653904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:21.5654403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:21.6113926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:21.6114413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:21.6116156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:21.6116663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:22.1222776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:51:22.1223338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:51:22.1224143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:22.1224854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:22.1230442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:51:22.1231337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:51:22.1324529Z STAGE:2022-09-27 16:51:22 127732:127732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:22.1325868Z STAGE:2022-09-27 16:51:22 127733:127733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:22.1341852Z STAGE:2022-09-27 16:51:22 127732:127732 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:51:22 127733:127733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:22.1342273Z 
2022-09-27T16:51:22.1360011Z STAGE:2022-09-27 16:51:22 127733:127733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:22.1370019Z STAGE:2022-09-27 16:51:22 127732:127732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:22.1385222Z STAGE:2022-09-27 16:51:22 127732:127732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:22.1385822Z STAGE:2022-09-27 16:51:22 127733:127733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:22.4832841Z ok (4.063s)
2022-09-27T16:51:22.4833057Z 
2022-09-27T16:51:22.4833450Z ----------------------------------------------------------------------
2022-09-27T16:51:22.4833778Z Ran 1 test in 4.063s
2022-09-27T16:51:22.4833949Z 
2022-09-27T16:51:22.4834053Z OK
2022-09-27T16:51:22.4834191Z 
2022-09-27T16:51:22.4834326Z Generating XML reports...
2022-09-27T16:51:22.4869512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165118.xml
2022-09-27T16:51:24.4946695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:24.4947223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:24.4948164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:24.4948655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:25.0015038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:25.0038044Z 
2022-09-27T16:51:25.0038262Z Running tests...
2022-09-27T16:51:25.0038702Z ----------------------------------------------------------------------
2022-09-27T16:51:26.5618809Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:51:26.5809998Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127845
2022-09-27T16:51:26.5816417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127846
2022-09-27T16:51:28.2065501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:28.2066337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:28.2067261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:28.2067766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:28.2179341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:28.2179822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:28.2183483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:28.2183979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:28.7471879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:51:28.7472469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:51:28.7473288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:28.7474001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:28.7581997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:51:28.7647329Z STAGE:2022-09-27 16:51:28 127846:127846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:28.8482314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:51:28.8546477Z STAGE:2022-09-27 16:51:28 127845:127845 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:28.8559565Z STAGE:2022-09-27 16:51:28 127845:127845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:28.8560406Z STAGE:2022-09-27 16:51:28 127846:127846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:28.8580205Z STAGE:2022-09-27 16:51:28 127845:127845 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-09-27 16:51:28 127846:127846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:28.8580603Z 
2022-09-27T16:51:28.8592201Z STAGE:2022-09-27 16:51:28 127845:127845 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:28.8592804Z STAGE:2022-09-27 16:51:28 127846:127846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:29.1878435Z ok (4.184s)
2022-09-27T16:51:29.1878671Z 
2022-09-27T16:51:29.1879046Z ----------------------------------------------------------------------
2022-09-27T16:51:29.1879388Z Ran 1 test in 4.184s
2022-09-27T16:51:29.1879556Z 
2022-09-27T16:51:29.1879652Z OK
2022-09-27T16:51:29.1879798Z 
2022-09-27T16:51:29.1879934Z Generating XML reports...
2022-09-27T16:51:29.1914671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165124.xml
2022-09-27T16:51:31.1918443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:31.1918961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:31.1919867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:31.1920972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:31.6903795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:31.6927190Z 
2022-09-27T16:51:31.6927442Z Running tests...
2022-09-27T16:51:31.6927882Z ----------------------------------------------------------------------
2022-09-27T16:51:33.2115489Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:51:33.2309022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127958
2022-09-27T16:51:33.2314904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127959
2022-09-27T16:51:34.8896565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:34.8897077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:34.8898413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:34.8898886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:34.9191077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:34.9191548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:34.9195060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:34.9195534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:35.4439038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:51:35.4439836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:51:35.4440879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:35.4441568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:35.4549596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:51:35.5450306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:51:35.9377523Z ok (4.245s)
2022-09-27T16:51:35.9377720Z 
2022-09-27T16:51:35.9378107Z ----------------------------------------------------------------------
2022-09-27T16:51:35.9378448Z Ran 1 test in 4.245s
2022-09-27T16:51:35.9378618Z 
2022-09-27T16:51:35.9378718Z OK
2022-09-27T16:51:35.9378867Z 
2022-09-27T16:51:35.9378978Z Generating XML reports...
2022-09-27T16:51:35.9414245Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165131.xml
2022-09-27T16:51:37.9010733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:37.9011258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:37.9012824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:37.9013511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:38.4085886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:38.4109825Z 
2022-09-27T16:51:38.4110329Z Running tests...
2022-09-27T16:51:38.4110761Z ----------------------------------------------------------------------
2022-09-27T16:51:39.9797385Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:51:39.9994905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128067
2022-09-27T16:51:40.0001669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128068
2022-09-27T16:51:41.6070062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:41.6070571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:41.6072579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:41.6073057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:41.6194941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:41.6195424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:41.6198659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:41.6199126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:42.1431115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:51:42.1431647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:51:42.1432406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:42.1433109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:42.1540361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:51:42.1606246Z STAGE:2022-09-27 16:51:42 128068:128068 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:42.2441358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:51:42.2507734Z STAGE:2022-09-27 16:51:42 128067:128067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:42.2521785Z STAGE:2022-09-27 16:51:42 128067:128067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:42.2522384Z STAGE:2022-09-27 16:51:42 128068:128068 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:42.2544687Z STAGE:2022-09-27 16:51:42 128068:128068 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:42.2545287Z STAGE:2022-09-27 16:51:42 128067:128067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:42.2558574Z STAGE:2022-09-27 16:51:42 128067:128067 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:51:42 128068:128068 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:42.2558992Z 
2022-09-27T16:51:42.6065087Z ok (4.195s)
2022-09-27T16:51:42.6065310Z 
2022-09-27T16:51:42.6065675Z ----------------------------------------------------------------------
2022-09-27T16:51:42.6066019Z Ran 1 test in 4.195s
2022-09-27T16:51:42.6066190Z 
2022-09-27T16:51:42.6066283Z OK
2022-09-27T16:51:42.6066418Z 
2022-09-27T16:51:42.6066549Z Generating XML reports...
2022-09-27T16:51:42.6102937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165138.xml
2022-09-27T16:51:44.5879862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:44.5880702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:44.5881779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:44.5882280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:45.0786854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:45.0808912Z 
2022-09-27T16:51:45.0809056Z Running tests...
2022-09-27T16:51:45.0809815Z ----------------------------------------------------------------------
2022-09-27T16:51:45.0827934Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:51:45.0828248Z 
2022-09-27T16:51:45.0828535Z ----------------------------------------------------------------------
2022-09-27T16:51:45.0828869Z Ran 1 test in 0.002s
2022-09-27T16:51:45.0829018Z 
2022-09-27T16:51:45.0829134Z OK (skipped=1)
2022-09-27T16:51:45.0829289Z 
2022-09-27T16:51:45.0829657Z Generating XML reports...
2022-09-27T16:51:45.0862694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165145.xml
2022-09-27T16:51:46.9199639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:46.9201269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:46.9201869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:46.9202356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:47.4244916Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:47.4268156Z 
2022-09-27T16:51:47.4268533Z Running tests...
2022-09-27T16:51:47.4268974Z ----------------------------------------------------------------------
2022-09-27T16:51:47.4287433Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-09-27T16:51:47.4287763Z 
2022-09-27T16:51:47.4288049Z ----------------------------------------------------------------------
2022-09-27T16:51:47.4288653Z Ran 1 test in 0.002s
2022-09-27T16:51:47.4288820Z 
2022-09-27T16:51:47.4288931Z OK (skipped=1)
2022-09-27T16:51:47.4289089Z 
2022-09-27T16:51:47.4289217Z Generating XML reports...
2022-09-27T16:51:47.4322537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165147.xml
2022-09-27T16:51:49.2798141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:49.2798652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:49.2800899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:49.2801699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:49.7773405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:49.7794838Z 
2022-09-27T16:51:49.7794982Z Running tests...
2022-09-27T16:51:49.7795666Z ----------------------------------------------------------------------
2022-09-27T16:51:51.3028953Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:51:51.3220049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128246
2022-09-27T16:51:51.3226487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128247
2022-09-27T16:51:52.8973862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:52.8974347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:52.8975832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:52.8976320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:52.9125367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:52.9125817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:52.9129678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:52.9130155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:53.4388815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:51:53.4389400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:51:53.4390389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:53.4391135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:51:53.4496660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:51:53.5400614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:51:53.5511860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:51:53.5512661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:51:53.5513375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:51:53.5514535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:51:53.5607537Z STAGE:2022-09-27 16:51:53 128246:128246 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:53.5608405Z STAGE:2022-09-27 16:51:53 128247:128247 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:53.5621404Z STAGE:2022-09-27 16:51:53 128247:128247 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:51:53 128246:128246 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:53.5621940Z 
2022-09-27T16:51:53.5641786Z STAGE:2022-09-27 16:51:53 128246:128246 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:53.5642748Z STAGE:2022-09-27 16:51:53 128247:128247 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:51:53.5654848Z STAGE:2022-09-27 16:51:53 128247:128247 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-09-27 16:51:53 128246:128246 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:51:53.5655257Z 
2022-09-27T16:51:53.9290044Z ok (4.149s)
2022-09-27T16:51:53.9290248Z 
2022-09-27T16:51:53.9290650Z ----------------------------------------------------------------------
2022-09-27T16:51:53.9291002Z Ran 1 test in 4.149s
2022-09-27T16:51:53.9291152Z 
2022-09-27T16:51:53.9291253Z OK
2022-09-27T16:51:53.9291390Z 
2022-09-27T16:51:53.9291523Z Generating XML reports...
2022-09-27T16:51:53.9327714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165149.xml
2022-09-27T16:51:55.9093334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:55.9094146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:55.9098417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:55.9099275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:56.4191406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:51:56.4215117Z 
2022-09-27T16:51:56.4215523Z Running tests...
2022-09-27T16:51:56.4216021Z ----------------------------------------------------------------------
2022-09-27T16:51:57.9758298Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:51:57.9955420Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128365
2022-09-27T16:51:57.9961847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128366
2022-09-27T16:51:59.5463419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:59.5463937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:59.5466283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:59.5467080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:51:59.6110265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:51:59.6110774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:51:59.6113183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:51:59.6113679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:00.1307053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:52:00.1307617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:52:00.1308407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:00.1309130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:00.1315503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:52:00.1316221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:52:00.5019698Z skip: Skipped due to small world size. (4.080s)
2022-09-27T16:52:00.5019955Z 
2022-09-27T16:52:00.5020334Z ----------------------------------------------------------------------
2022-09-27T16:52:00.5020686Z Ran 1 test in 4.080s
2022-09-27T16:52:00.5020857Z 
2022-09-27T16:52:00.5020947Z OK (skipped=1)
2022-09-27T16:52:00.5021104Z 
2022-09-27T16:52:00.5021236Z Generating XML reports...
2022-09-27T16:52:00.5065580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165156.xml
2022-09-27T16:52:02.5288910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:02.5289432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:02.5291436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:02.5291946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:03.0217132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:03.0239783Z 
2022-09-27T16:52:03.0239926Z Running tests...
2022-09-27T16:52:03.0240708Z ----------------------------------------------------------------------
2022-09-27T16:52:04.5677819Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:52:04.5869708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128474
2022-09-27T16:52:04.5877042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128475
2022-09-27T16:52:06.1692597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:06.1693133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:06.1694253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:06.1694756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:06.2277707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:06.2278202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:06.2280659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:06.2281394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:06.7251545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:52:06.7252339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:52:06.7253164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:06.7253871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:06.7359367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:52:06.8262714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:52:07.1939914Z ok (4.170s)
2022-09-27T16:52:07.1940278Z 
2022-09-27T16:52:07.1941013Z ----------------------------------------------------------------------
2022-09-27T16:52:07.1941715Z Ran 1 test in 4.170s
2022-09-27T16:52:07.1941925Z 
2022-09-27T16:52:07.1942022Z OK
2022-09-27T16:52:07.1942156Z 
2022-09-27T16:52:07.1942297Z Generating XML reports...
2022-09-27T16:52:07.1978416Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165203.xml
2022-09-27T16:52:09.2097908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:09.2098458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:09.2100431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:09.2100928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:09.7132277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:09.7155584Z 
2022-09-27T16:52:09.7155954Z Running tests...
2022-09-27T16:52:09.7156463Z ----------------------------------------------------------------------
2022-09-27T16:52:11.2684256Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:52:11.2882324Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128583
2022-09-27T16:52:11.2888860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128584
2022-09-27T16:52:12.9294086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:12.9294632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:12.9295870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:12.9296362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:12.9297179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:12.9297674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:12.9301541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:12.9302040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:13.4690043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:52:13.4690613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:52:13.4691401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:13.4692109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:13.4698123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:52:13.4698909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:52:13.7947353Z ok (4.079s)
2022-09-27T16:52:13.7947577Z 
2022-09-27T16:52:13.7947964Z ----------------------------------------------------------------------
2022-09-27T16:52:13.7948287Z Ran 1 test in 4.079s
2022-09-27T16:52:13.7948458Z 
2022-09-27T16:52:13.7948553Z OK
2022-09-27T16:52:13.7948696Z 
2022-09-27T16:52:13.7948832Z Generating XML reports...
2022-09-27T16:52:13.8002614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165209.xml
2022-09-27T16:52:15.7731480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:15.7731990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:15.7733376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:15.7733856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:16.2641717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:16.2664968Z 
2022-09-27T16:52:16.2665483Z Running tests...
2022-09-27T16:52:16.2665997Z ----------------------------------------------------------------------
2022-09-27T16:52:17.7902408Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:52:17.8094727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128692
2022-09-27T16:52:17.8101239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128693
2022-09-27T16:52:19.4501724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:19.4502306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:19.4503541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:19.4504034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:19.4711632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:19.4712116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:19.4715205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:19.4715700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:20.0099788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:52:20.0100512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:52:20.0101335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:20.0102049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:20.0107931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:52:20.0108424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:52:20.4164339Z ok (4.150s)
2022-09-27T16:52:20.4164555Z 
2022-09-27T16:52:20.4164949Z ----------------------------------------------------------------------
2022-09-27T16:52:20.4165290Z Ran 1 test in 4.150s
2022-09-27T16:52:20.4165445Z 
2022-09-27T16:52:20.4165542Z OK
2022-09-27T16:52:20.4165679Z 
2022-09-27T16:52:20.4165813Z Generating XML reports...
2022-09-27T16:52:20.4201151Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165216.xml
2022-09-27T16:52:22.3753074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:22.3753888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:22.3754715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:22.3755210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:22.8818540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:22.8841848Z 
2022-09-27T16:52:22.8842221Z Running tests...
2022-09-27T16:52:22.8842700Z ----------------------------------------------------------------------
2022-09-27T16:52:24.4436714Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:52:24.4633541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128801
2022-09-27T16:52:24.4639758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128802
2022-09-27T16:52:26.0402488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:26.0403519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:26.0404470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:26.0404964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:26.0671987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:26.0672472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:26.0675962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:26.0676469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:26.5888886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:52:26.5889469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:52:26.5890245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:26.5890946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:26.5896497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:52:26.5896999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:52:26.5991905Z STAGE:2022-09-27 16:52:26 128802:128802 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:52:26.5992547Z STAGE:2022-09-27 16:52:26 128801:128801 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:52:26.6015717Z STAGE:2022-09-27 16:52:26 128801:128801 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:52:26.6016344Z STAGE:2022-09-27 16:52:26 128802:128802 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:52:26.9700687Z ok (4.086s)
2022-09-27T16:52:26.9700906Z 
2022-09-27T16:52:26.9701301Z ----------------------------------------------------------------------
2022-09-27T16:52:26.9701644Z Ran 1 test in 4.086s
2022-09-27T16:52:26.9701819Z 
2022-09-27T16:52:26.9701902Z OK
2022-09-27T16:52:26.9702039Z 
2022-09-27T16:52:26.9702174Z Generating XML reports...
2022-09-27T16:52:26.9739174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165222.xml
2022-09-27T16:52:28.9309443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:28.9310026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:28.9311775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:28.9312272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:29.4237191Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:29.4259764Z 
2022-09-27T16:52:29.4260048Z Running tests...
2022-09-27T16:52:29.4260476Z ----------------------------------------------------------------------
2022-09-27T16:52:30.9509120Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:52:30.9701169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128914
2022-09-27T16:52:30.9707591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128915
2022-09-27T16:52:32.6105633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:32.6106639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:32.6107672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:32.6108201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:32.6192262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:32.6192725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:32.6196171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:32.6196664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:33.1488070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:52:33.1488617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:52:33.1489423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:33.1490124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:33.1496003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:52:33.1496484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:52:33.1592215Z STAGE:2022-09-27 16:52:33 128914:128914 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:52:33.1592821Z STAGE:2022-09-27 16:52:33 128915:128915 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:52:33.1617004Z STAGE:2022-09-27 16:52:33 128914:128914 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:52:33.1617598Z STAGE:2022-09-27 16:52:33 128915:128915 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:52:33.5769890Z ok (4.151s)
2022-09-27T16:52:33.5770108Z 
2022-09-27T16:52:33.5770522Z ----------------------------------------------------------------------
2022-09-27T16:52:33.5770850Z Ran 1 test in 4.151s
2022-09-27T16:52:33.5771023Z 
2022-09-27T16:52:33.5771118Z OK
2022-09-27T16:52:33.5771254Z 
2022-09-27T16:52:33.5771389Z Generating XML reports...
2022-09-27T16:52:33.5806640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165229.xml
2022-09-27T16:52:35.5763589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:35.5764388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:35.5765635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:35.5766135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:36.0823632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:36.0846977Z 
2022-09-27T16:52:36.0847337Z Running tests...
2022-09-27T16:52:36.0847807Z ----------------------------------------------------------------------
2022-09-27T16:52:37.6549846Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:52:37.6746393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129027
2022-09-27T16:52:37.6752535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129028
2022-09-27T16:52:39.2807076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:39.2807587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:39.2809544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:39.2810372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:39.2934129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:39.2934579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:39.2937853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:39.2938350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:39.8138137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:52:39.8138703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:52:39.8139495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:39.8140216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:39.8246140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:52:39.8310874Z STAGE:2022-09-27 16:52:39 129028:129028 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:52:39.9151746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:52:39.9223824Z STAGE:2022-09-27 16:52:39 129027:129027 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:52:39.9249774Z STAGE:2022-09-27 16:52:39 129028:129028 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:52:39.9257972Z STAGE:2022-09-27 16:52:39 129027:129027 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:52:40.2820353Z ok (4.197s)
2022-09-27T16:52:40.2820643Z 
2022-09-27T16:52:40.2821041Z ----------------------------------------------------------------------
2022-09-27T16:52:40.2821383Z Ran 1 test in 4.197s
2022-09-27T16:52:40.2821549Z 
2022-09-27T16:52:40.2821645Z OK
2022-09-27T16:52:40.2821768Z 
2022-09-27T16:52:40.2821906Z Generating XML reports...
2022-09-27T16:52:40.2866153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165236.xml
2022-09-27T16:52:42.2921614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:42.2922174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:42.2924415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:42.2924937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:42.7854325Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:42.7875921Z 
2022-09-27T16:52:42.7876347Z Running tests...
2022-09-27T16:52:42.7877040Z ----------------------------------------------------------------------
2022-09-27T16:52:42.7894605Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:52:42.7894899Z 
2022-09-27T16:52:42.7895180Z ----------------------------------------------------------------------
2022-09-27T16:52:42.7895515Z Ran 1 test in 0.002s
2022-09-27T16:52:42.7895664Z 
2022-09-27T16:52:42.7895784Z OK (skipped=1)
2022-09-27T16:52:42.7895941Z 
2022-09-27T16:52:42.7896071Z Generating XML reports...
2022-09-27T16:52:42.7929271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165242.xml
2022-09-27T16:52:44.6217671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:44.6218734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:44.6219721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:44.6220209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:45.1206295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:45.1229541Z 
2022-09-27T16:52:45.1229977Z Running tests...
2022-09-27T16:52:45.1230468Z ----------------------------------------------------------------------
2022-09-27T16:52:45.1248513Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:52:45.1248836Z 
2022-09-27T16:52:45.1249134Z ----------------------------------------------------------------------
2022-09-27T16:52:45.1249451Z Ran 1 test in 0.002s
2022-09-27T16:52:45.1249623Z 
2022-09-27T16:52:45.1249733Z OK (skipped=1)
2022-09-27T16:52:45.1249898Z 
2022-09-27T16:52:45.1251537Z Generating XML reports...
2022-09-27T16:52:45.1284826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165245.xml
2022-09-27T16:52:46.9913369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:46.9913883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:46.9915413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:46.9915902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:47.4939056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:47.4962851Z 
2022-09-27T16:52:47.4963334Z Running tests...
2022-09-27T16:52:47.4963825Z ----------------------------------------------------------------------
2022-09-27T16:52:47.4984119Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-09-27T16:52:47.4984436Z 
2022-09-27T16:52:47.4984810Z ----------------------------------------------------------------------
2022-09-27T16:52:47.4985330Z Ran 1 test in 0.002s
2022-09-27T16:52:47.4985484Z 
2022-09-27T16:52:47.4985600Z OK (skipped=1)
2022-09-27T16:52:47.4985755Z 
2022-09-27T16:52:47.4985882Z Generating XML reports...
2022-09-27T16:52:47.5020247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165247.xml
2022-09-27T16:52:49.3491680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:49.3492504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:49.3493643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:49.3494145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:49.8556908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:49.8580027Z 
2022-09-27T16:52:49.8580480Z Running tests...
2022-09-27T16:52:49.8580976Z ----------------------------------------------------------------------
2022-09-27T16:52:51.4108462Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:52:51.4303818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129239
2022-09-27T16:52:51.4310519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129240
2022-09-27T16:52:53.0119339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:53.0119849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:53.0121456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:53.0121972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:53.0306333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:53.0306818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:53.0309991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:53.0310480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:53.5547618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:52:53.5548177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:52:53.5548949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:53.5549665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:52:53.5555929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:52:53.5556422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:52:53.5651514Z STAGE:2022-09-27 16:52:53 129240:129240 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:52:53.5652160Z STAGE:2022-09-27 16:52:53 129239:129239 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:52:53.5675708Z STAGE:2022-09-27 16:52:53 129240:129240 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:52:53.5684204Z STAGE:2022-09-27 16:52:53 129239:129239 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:52:53.9369750Z ok (4.079s)
2022-09-27T16:52:53.9369967Z 
2022-09-27T16:52:53.9370351Z ----------------------------------------------------------------------
2022-09-27T16:52:53.9370675Z Ran 1 test in 4.079s
2022-09-27T16:52:53.9370842Z 
2022-09-27T16:52:53.9370941Z OK
2022-09-27T16:52:53.9371079Z 
2022-09-27T16:52:53.9371216Z Generating XML reports...
2022-09-27T16:52:53.9415041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165249.xml
2022-09-27T16:52:55.9273084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:55.9273597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:55.9275326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:55.9275823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:56.4409166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:52:56.4432321Z 
2022-09-27T16:52:56.4432605Z Running tests...
2022-09-27T16:52:56.4433467Z ----------------------------------------------------------------------
2022-09-27T16:52:58.0082340Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:52:58.0279510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129352
2022-09-27T16:52:58.0286384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129353
2022-09-27T16:52:59.6547204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:59.6547733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:59.6548975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:59.6549792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:52:59.6895095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:52:59.6895550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:52:59.6899030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:52:59.6899518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:00.2003925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:00.2004505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:00.2005298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:00.2006018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:00.2112678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:00.3014627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:00.6346746Z ok (4.191s)
2022-09-27T16:53:00.6346958Z 
2022-09-27T16:53:00.6347335Z ----------------------------------------------------------------------
2022-09-27T16:53:00.6347671Z Ran 1 test in 4.191s
2022-09-27T16:53:00.6347822Z 
2022-09-27T16:53:00.6347920Z OK
2022-09-27T16:53:00.6348056Z 
2022-09-27T16:53:00.6348192Z Generating XML reports...
2022-09-27T16:53:00.6383865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165256.xml
2022-09-27T16:53:02.6227885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:02.6228427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:02.6229779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:02.6230269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:03.1401642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:53:03.1425244Z 
2022-09-27T16:53:03.1425476Z Running tests...
2022-09-27T16:53:03.1425914Z ----------------------------------------------------------------------
2022-09-27T16:53:04.6803054Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:53:04.7001132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129461
2022-09-27T16:53:04.7007222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129462
2022-09-27T16:53:06.3393099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:06.3393796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:06.3394593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:06.3395081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:06.3411897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:06.3412368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:06.3415954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:06.3416443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:06.8900786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:06.8901612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:06.8902405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:06.8903114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:06.9009180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:06.9073712Z STAGE:2022-09-27 16:53:06 129462:129462 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:53:06.9910976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:06.9975372Z STAGE:2022-09-27 16:53:06 129461:129461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:53:06.9991092Z STAGE:2022-09-27 16:53:06 129461:129461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:53:06.9993937Z STAGE:2022-09-27 16:53:06 129462:129462 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:53:07.4075394Z ok (4.265s)
2022-09-27T16:53:07.4075717Z 
2022-09-27T16:53:07.4076500Z ----------------------------------------------------------------------
2022-09-27T16:53:07.4077067Z Ran 1 test in 4.265s
2022-09-27T16:53:07.4077236Z 
2022-09-27T16:53:07.4077331Z OK
2022-09-27T16:53:07.4077449Z 
2022-09-27T16:53:07.4077580Z Generating XML reports...
2022-09-27T16:53:07.4113654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165303.xml
2022-09-27T16:53:09.4122362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:09.4122899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:09.4125170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:09.4125663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:09.9039078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:53:09.9061550Z 
2022-09-27T16:53:09.9061712Z Running tests...
2022-09-27T16:53:09.9062158Z ----------------------------------------------------------------------
2022-09-27T16:53:11.4543783Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:53:11.4736630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129574
2022-09-27T16:53:11.4743090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129575
2022-09-27T16:53:13.0937209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:13.0937759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:13.0938709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:13.0939188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:13.1004487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:13.1004944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:13.1008386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:13.1008886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:13.6252231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:13.6252784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:13.6253848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:13.6254554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:13.6360439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:13.6425483Z STAGE:2022-09-27 16:53:13 129575:129575 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:53:13.7264636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:13.7329412Z STAGE:2022-09-27 16:53:13 129574:129574 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-09-27T16:53:13.7346328Z STAGE:2022-09-27 16:53:13 129574:129574 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:53:13.7348810Z STAGE:2022-09-27 16:53:13 129575:129575 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-09-27T16:53:14.0804459Z ok (4.174s)
2022-09-27T16:53:14.0804659Z 
2022-09-27T16:53:14.0805059Z ----------------------------------------------------------------------
2022-09-27T16:53:14.0805402Z Ran 1 test in 4.174s
2022-09-27T16:53:14.0805575Z 
2022-09-27T16:53:14.0805667Z OK
2022-09-27T16:53:14.0805809Z 
2022-09-27T16:53:14.0805926Z Generating XML reports...
2022-09-27T16:53:14.0841481Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165309.xml
2022-09-27T16:53:16.0673684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:16.0674223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:16.0676203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:16.0676696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:16.5591573Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:53:16.5614662Z 
2022-09-27T16:53:16.5614805Z Running tests...
2022-09-27T16:53:16.5615704Z ----------------------------------------------------------------------
2022-09-27T16:53:18.1217785Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:53:18.1409975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129687
2022-09-27T16:53:18.1416601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129688
2022-09-27T16:53:19.7047099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:19.7047999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:19.7049175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:19.7049676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:19.7285538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:19.7286011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:19.7289538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:19.7290014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:20.2540489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:20.2541040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:20.2541845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:20.2542827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:20.2549094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:20.2549594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:20.6485116Z ok (4.087s)
2022-09-27T16:53:20.6485368Z 
2022-09-27T16:53:20.6485767Z ----------------------------------------------------------------------
2022-09-27T16:53:20.6486122Z Ran 1 test in 4.087s
2022-09-27T16:53:20.6486294Z 
2022-09-27T16:53:20.6486394Z OK
2022-09-27T16:53:20.6486513Z 
2022-09-27T16:53:20.6486671Z Generating XML reports...
2022-09-27T16:53:20.6522369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165316.xml
2022-09-27T16:53:22.6736061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:22.6736605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:22.6737218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:22.6737689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:23.1641204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:53:23.1663630Z 
2022-09-27T16:53:23.1663903Z Running tests...
2022-09-27T16:53:23.1664351Z ----------------------------------------------------------------------
2022-09-27T16:53:24.6986575Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:53:24.7179116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129886
2022-09-27T16:53:24.7185468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129887
2022-09-27T16:53:26.3112437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:26.3112934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:26.3114285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:26.3114773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:26.3211099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:26.3211847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:26.3214935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:26.3215433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:26.8535383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:26.8535929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:26.8536711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:26.8537420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:26.8645828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:26.9550386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:28.7275063Z ok (5.561s)
2022-09-27T16:53:28.7275305Z 
2022-09-27T16:53:28.7275717Z ----------------------------------------------------------------------
2022-09-27T16:53:28.7276370Z Ran 1 test in 5.561s
2022-09-27T16:53:28.7276538Z 
2022-09-27T16:53:28.7276636Z OK
2022-09-27T16:53:28.7276772Z 
2022-09-27T16:53:28.7276897Z Generating XML reports...
2022-09-27T16:53:28.7312113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165323.xml
2022-09-27T16:53:30.7418091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:30.7418589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:30.7419838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:30.7420335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:31.2293222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:53:31.2320615Z 
2022-09-27T16:53:31.2320736Z Running tests...
2022-09-27T16:53:31.2321414Z ----------------------------------------------------------------------
2022-09-27T16:53:32.7753706Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:53:32.7943824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130087
2022-09-27T16:53:32.7950630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130088
2022-09-27T16:53:34.4057492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:34.4058061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:34.4059526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:34.4060018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:34.4545694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:34.4546166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:34.4549740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:34.4550227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:34.9805725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:34.9806403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:34.9807391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:34.9808122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:34.9915515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:35.0819695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:37.3046296Z ok (6.072s)
2022-09-27T16:53:37.3046638Z 
2022-09-27T16:53:37.3047045Z ----------------------------------------------------------------------
2022-09-27T16:53:37.3047373Z Ran 1 test in 6.072s
2022-09-27T16:53:37.3047541Z 
2022-09-27T16:53:37.3047636Z OK
2022-09-27T16:53:37.3047772Z 
2022-09-27T16:53:37.3047903Z Generating XML reports...
2022-09-27T16:53:37.3083583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165331.xml
2022-09-27T16:53:39.3067427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:39.3067928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:39.3069379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:39.3069870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:39.7976859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:53:39.7999066Z 
2022-09-27T16:53:39.7999228Z Running tests...
2022-09-27T16:53:39.7999977Z ----------------------------------------------------------------------
2022-09-27T16:53:41.3308786Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:53:41.3500753Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130202
2022-09-27T16:53:41.3508211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130203
2022-09-27T16:53:42.9211398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:42.9211945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:42.9213856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:42.9214351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:42.9542896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:42.9543375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:42.9546683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:42.9547191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:43.4838673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:43.4839235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:43.4840048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:43.4841040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:43.4847266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:43.4847762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:43.4948429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0wm3_hq
2022-09-27T16:53:43.4949180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2ow6u25
2022-09-27T16:53:43.4951102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0wm3_hq/_remote_module_non_scriptable.py
2022-09-27T16:53:43.4951666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2ow6u25/_remote_module_non_scriptable.py
2022-09-27T16:53:43.8569359Z ok (4.057s)
2022-09-27T16:53:43.8569597Z 
2022-09-27T16:53:43.8570002Z ----------------------------------------------------------------------
2022-09-27T16:53:43.8570323Z Ran 1 test in 4.057s
2022-09-27T16:53:43.8570493Z 
2022-09-27T16:53:43.8570587Z OK
2022-09-27T16:53:43.8570723Z 
2022-09-27T16:53:43.8570856Z Generating XML reports...
2022-09-27T16:53:43.8605705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165339.xml
2022-09-27T16:53:45.8346034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:45.8346574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:45.8347981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:45.8348777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:46.3521444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:53:46.3544664Z 
2022-09-27T16:53:46.3545325Z Running tests...
2022-09-27T16:53:46.3545913Z ----------------------------------------------------------------------
2022-09-27T16:53:47.9178781Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:53:47.9368838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130315
2022-09-27T16:53:47.9375802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130316
2022-09-27T16:53:49.5915294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:49.5916289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:49.5917448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:49.5918393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:49.5948288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:49.5948776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:49.5952274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:49.5952757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:50.1237693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:50.1238743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:50.1240726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:50.1242063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:50.1246522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:50.1247442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:51.4438308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna8ni0bb
2022-09-27T16:53:51.4439468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna8ni0bb/_remote_module_non_scriptable.py
2022-09-27T16:53:51.4717408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpplq3_ytk
2022-09-27T16:53:51.4719640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpplq3_ytk/_remote_module_non_scriptable.py
2022-09-27T16:53:51.9461860Z ok (5.591s)
2022-09-27T16:53:51.9462059Z 
2022-09-27T16:53:51.9462693Z ----------------------------------------------------------------------
2022-09-27T16:53:51.9463339Z Ran 1 test in 5.592s
2022-09-27T16:53:51.9463521Z 
2022-09-27T16:53:51.9463616Z OK
2022-09-27T16:53:51.9463736Z 
2022-09-27T16:53:51.9463866Z Generating XML reports...
2022-09-27T16:53:51.9499016Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165346.xml
2022-09-27T16:53:53.9195386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:53.9195921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:53.9197089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:53.9197580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:54.4150807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:53:54.4173270Z 
2022-09-27T16:53:54.4173661Z Running tests...
2022-09-27T16:53:54.4174157Z ----------------------------------------------------------------------
2022-09-27T16:53:55.9442018Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:53:55.9637800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130426
2022-09-27T16:53:55.9644179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130427
2022-09-27T16:53:57.4944797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:57.4945345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:57.4946576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:57.4947070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:57.5179731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:53:57.5180210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:53:57.5184340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:53:57.5184830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:53:58.0436384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:53:58.0436943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:53:58.0437746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:58.0438464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:53:58.0445295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:53:58.0445795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:53:59.3651686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf48o_gu0
2022-09-27T16:53:59.3652317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf48o_gu0/_remote_module_non_scriptable.py
2022-09-27T16:53:59.3889920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1a3l75c4
2022-09-27T16:53:59.3892590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1a3l75c4/_remote_module_non_scriptable.py
2022-09-27T16:53:59.8345861Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:53:59.8373804Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-09-27T16:54:00.3758190Z ok (5.958s)
2022-09-27T16:54:00.3758406Z 
2022-09-27T16:54:00.3759077Z ----------------------------------------------------------------------
2022-09-27T16:54:00.3759414Z Ran 1 test in 5.958s
2022-09-27T16:54:00.3759585Z 
2022-09-27T16:54:00.3759682Z OK
2022-09-27T16:54:00.3760665Z 
2022-09-27T16:54:00.3760929Z Generating XML reports...
2022-09-27T16:54:00.3796667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165354.xml
2022-09-27T16:54:02.4065562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:54:02.4066091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:54:02.4067184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:54:02.4067680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:54:02.9238100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:54:02.9262143Z 
2022-09-27T16:54:02.9262554Z Running tests...
2022-09-27T16:54:02.9263076Z ----------------------------------------------------------------------
2022-09-27T16:54:04.4805861Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:54:04.5002120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130541
2022-09-27T16:54:04.5008855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130542
2022-09-27T16:54:06.1266407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:54:06.1266921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:54:06.1267998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:54:06.1268490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:54:06.1460001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:54:06.1460457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:54:06.1464120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:54:06.1464608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:54:06.6874045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:54:06.6874639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:54:06.6875661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:54:06.6876401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:54:06.6981892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:54:06.7884290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:54:06.8094015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:54:06.8094542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:54:06.8095238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:54:06.8095928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:54:06.8303383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:54:06.8303899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:54:06.8304756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:54:06.8305439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:54:08.1368466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkufbzgc
2022-09-27T16:54:08.1369066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkufbzgc/_remote_module_non_scriptable.py
2022-09-27T16:54:08.1504674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70_nddhk
2022-09-27T16:54:08.1507646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70_nddhk/_remote_module_non_scriptable.py
2022-09-27T16:54:13.6184463Z ok (10.692s)
2022-09-27T16:54:13.6184685Z 
2022-09-27T16:54:13.6185064Z ----------------------------------------------------------------------
2022-09-27T16:54:13.6185409Z Ran 1 test in 10.692s
2022-09-27T16:54:13.6185576Z 
2022-09-27T16:54:13.6185676Z OK
2022-09-27T16:54:13.6185814Z 
2022-09-27T16:54:13.6185950Z Generating XML reports...
2022-09-27T16:54:13.6221881Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165402.xml
2022-09-27T16:54:15.6409616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:54:15.6410147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:54:15.6411219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:54:15.6411711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:54:16.1308062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-09-27T16:54:16.1330177Z 
2022-09-27T16:54:16.1330436Z Running tests...
2022-09-27T16:54:16.1330884Z ----------------------------------------------------------------------
2022-09-27T16:54:17.6623902Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-09-27T16:54:17.6815895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130664
2022-09-27T16:54:17.6822738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130665
2022-09-27T16:54:19.2561270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:54:19.2561803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:54:19.2563427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:54:19.2563934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:54:19.2937588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:123: UserWarning: loaded 45 slow tests
2022-09-27T16:54:19.2938076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-09-27T16:54:19.2940883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:127: UserWarning: loaded 261 disabled tests
2022-09-27T16:54:19.2941395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-09-27T16:54:19.8105332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-09-27T16:54:19.8105894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-09-27T16:54:19.8106684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:54:19.8107399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-09-27T16:54:19.8113664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-09-27T16:54:19.8114339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-09-27T16:54:19.8222244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-09-27T16:54:19.8222861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-09-27T16:54:19.8223554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:54:19.8224253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-09-27T16:54:19.8330446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-09-27T16:54:19.8331058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-09-27T16:54:19.8331754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:54:19.8332450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-09-27T16:54:21.1559326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltsihci_
2022-09-27T16:54:21.1560632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltsihci_/_remote_module_non_scriptable.py
2022-09-27T16:54:21.1680833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7wczl8v
2022-09-27T16:54:21.1684024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7wczl8v/_remote_module_non_scriptable.py
2022-09-27T16:54:26.7000042Z ok (10.567s)
2022-09-27T16:54:26.7000548Z 
2022-09-27T16:54:26.7000937Z ----------------------------------------------------------------------
2022-09-27T16:54:26.7001310Z Ran 1 test in 10.567s
2022-09-27T16:54:26.7001477Z 
2022-09-27T16:54:26.7001554Z OK
2022-09-27T16:54:26.7001689Z 
2022-09-27T16:54:26.7001822Z Generating XML reports...
2022-09-27T16:54:26.7037157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165416.xml
2022-09-27T16:54:27.3327948Z Running distributed/_shard/test_sharder ... [2022-09-27 16:54:27.332298]
2022-09-27T16:54:27.3328700Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:54:27.332380]
2022-09-27T16:54:29.1810496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf416e4ey
2022-09-27T16:54:29.1811379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf416e4ey/_remote_module_non_scriptable.py
2022-09-27T16:54:29.4427061Z Running distributed/fsdp/test_shard_utils ... [2022-09-27 16:54:29.442231]
2022-09-27T16:54:29.4427858Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:54:29.442330]
2022-09-27T16:54:31.5613470Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-09-27 16:54:31.560822]
2022-09-27T16:54:31.5614194Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-09-27 16:54:31.560906]
2022-09-27T16:54:34.1328502Z ============================= test session starts ==============================
2022-09-27T16:54:34.1329085Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:54:34.1458495Z cachedir: .pytest_cache
2022-09-27T16:54:34.1459337Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:54:34.1459793Z torch: 1.13.0a0+git52424e2
2022-09-27T16:54:34.1460365Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:54:34.1460893Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:54:34.1640632Z collecting ... 
2022-09-27T16:54:34.1641484Z collected 6 items                                                              
2022-09-27T16:54:34.1645195Z 
2022-09-27T16:54:34.1674671Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2022-09-27T16:54:34.1692256Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2022-09-27T16:54:34.1709325Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2022-09-27T16:54:34.1726635Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2022-09-27T16:54:34.1744267Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2022-09-27T16:54:34.1767798Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2022-09-27T16:54:34.1768397Z 
2022-09-27T16:54:34.1768721Z ============================== 6 passed in 0.04s ===============================
2022-09-27T16:54:34.4464388Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-09-27 16:54:34.445967]
2022-09-27T16:54:34.4465079Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-09-27 16:54:34.446046]
2022-09-27T16:54:36.5256117Z ============================= test session starts ==============================
2022-09-27T16:54:36.5256695Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:54:36.5352914Z cachedir: .pytest_cache
2022-09-27T16:54:36.5353764Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:54:36.5354219Z torch: 1.13.0a0+git52424e2
2022-09-27T16:54:36.5354555Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:54:36.5355098Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:54:36.5534207Z collecting ... 
2022-09-27T16:54:36.5534625Z collected 7 items                                                              
2022-09-27T16:54:36.5538741Z 
2022-09-27T16:54:36.5781380Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2022-09-27T16:54:36.5801276Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2022-09-27T16:54:36.5821967Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2022-09-27T16:54:36.5839973Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2022-09-27T16:54:36.5859835Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2022-09-27T16:54:36.5877727Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2022-09-27T16:54:36.5912217Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2022-09-27T16:54:36.5913675Z 
2022-09-27T16:54:36.5913987Z ============================== 7 passed in 0.07s ===============================
2022-09-27T16:54:36.8573133Z Running distributed/pipeline/sync/test_balance ... [2022-09-27 16:54:36.856823]
2022-09-27T16:54:36.8573820Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-09-27 16:54:36.856899]
2022-09-27T16:54:38.9221030Z ============================= test session starts ==============================
2022-09-27T16:54:38.9221948Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:54:38.9316024Z cachedir: .pytest_cache
2022-09-27T16:54:38.9316647Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:54:38.9317094Z torch: 1.13.0a0+git52424e2
2022-09-27T16:54:38.9317435Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:54:38.9317977Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:54:38.9866338Z collecting ... 
2022-09-27T16:54:38.9866772Z collected 18 items                                                             
2022-09-27T16:54:38.9871644Z 
2022-09-27T16:54:38.9902706Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  5%]
2022-09-27T16:54:38.9918319Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%]
2022-09-27T16:54:38.9934903Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%]
2022-09-27T16:54:38.9950446Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%]
2022-09-27T16:54:38.9961513Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%]
2022-09-27T16:54:38.9971969Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%]
2022-09-27T16:54:40.0018303Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%]
2022-09-27T16:54:41.2931566Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%]
2022-09-27T16:54:41.7145856Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%]
2022-09-27T16:54:41.7274025Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%]
2022-09-27T16:54:41.7303022Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%]
2022-09-27T16:54:41.7330179Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%]
2022-09-27T16:54:42.7368171Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%]
2022-09-27T16:54:43.7403101Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%]
2022-09-27T16:54:44.7436566Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 83%]
2022-09-27T16:54:45.7465277Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%]
2022-09-27T16:54:45.7496941Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%]
2022-09-27T16:54:45.7524556Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2022-09-27T16:54:45.7526148Z 
2022-09-27T16:54:45.7526639Z ======================== 16 passed, 2 skipped in 6.83s =========================
2022-09-27T16:54:46.2238867Z Running distributed/pipeline/sync/test_copy ... [2022-09-27 16:54:46.223348]
2022-09-27T16:54:46.2239544Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-09-27 16:54:46.223426]
2022-09-27T16:54:48.3161508Z ============================= test session starts ==============================
2022-09-27T16:54:48.3162106Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:54:48.3257214Z cachedir: .pytest_cache
2022-09-27T16:54:48.3257809Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:54:48.3258656Z torch: 1.13.0a0+git52424e2
2022-09-27T16:54:48.3259002Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:54:48.3259552Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:54:48.3679102Z collecting ... 
2022-09-27T16:54:48.3679522Z collected 5 items                                                              
2022-09-27T16:54:48.3684261Z 
2022-09-27T16:54:48.3739447Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2022-09-27T16:54:49.6423739Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED   [ 40%]
2022-09-27T16:54:50.0654022Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED   [ 60%]
2022-09-27T16:54:50.4008670Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED  [ 80%]
2022-09-27T16:54:50.4027599Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2022-09-27T16:54:50.4028650Z 
2022-09-27T16:54:50.4028984Z ============================== 5 passed in 2.09s ===============================
2022-09-27T16:54:50.7403402Z Running distributed/pipeline/sync/test_inplace ... [2022-09-27 16:54:50.739864]
2022-09-27T16:54:50.7404084Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-09-27 16:54:50.739942]
2022-09-27T16:54:52.7778674Z ============================= test session starts ==============================
2022-09-27T16:54:52.7779234Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:54:52.7872077Z cachedir: .pytest_cache
2022-09-27T16:54:52.7872803Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:54:52.7873416Z torch: 1.13.0a0+git52424e2
2022-09-27T16:54:52.7873867Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:54:52.7874416Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:54:52.8000507Z collecting ... 
2022-09-27T16:54:52.8001248Z collected 3 items                                                              
2022-09-27T16:54:52.8014997Z 
2022-09-27T16:54:52.9180022Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2022-09-27T16:54:52.9445091Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2022-09-27T16:54:52.9706647Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2022-09-27T16:54:52.9707524Z 
2022-09-27T16:54:52.9707895Z ========================= 1 passed, 2 xfailed in 0.19s =========================
2022-09-27T16:54:53.2288063Z Running distributed/pipeline/sync/test_pipe ... [2022-09-27 16:54:53.228323]
2022-09-27T16:54:53.2288707Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-09-27 16:54:53.228411]
2022-09-27T16:54:55.2987930Z ============================= test session starts ==============================
2022-09-27T16:54:55.2988524Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:54:55.3085887Z cachedir: .pytest_cache
2022-09-27T16:54:55.3086718Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:54:55.3087179Z torch: 1.13.0a0+git52424e2
2022-09-27T16:54:55.3087541Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:54:55.3088059Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:54:55.4505275Z collecting ... 
2022-09-27T16:54:55.4505715Z collected 56 items                                                             
2022-09-27T16:54:55.4509532Z 
2022-09-27T16:54:55.4553318Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2022-09-27T16:54:55.5117439Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2022-09-27T16:54:55.5375836Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2022-09-27T16:54:55.5636999Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2022-09-27T16:54:55.5890660Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2022-09-27T16:54:55.6171562Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2022-09-27T16:54:55.6440995Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2022-09-27T16:54:55.6735491Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2022-09-27T16:54:55.6991478Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2022-09-27T16:54:55.7256003Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2022-09-27T16:54:55.7530419Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2022-09-27T16:54:55.7813209Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2022-09-27T16:54:55.8079828Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2022-09-27T16:54:55.8238423Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2022-09-27T16:54:56.0433169Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2022-09-27T16:54:56.0619368Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2022-09-27T16:54:56.0797113Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2022-09-27T16:54:56.0965169Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2022-09-27T16:54:56.1138550Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2022-09-27T16:54:56.1298598Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2022-09-27T16:54:56.1457533Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2022-09-27T16:54:56.1631075Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2022-09-27T16:54:56.1868940Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2022-09-27T16:54:56.2193542Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2022-09-27T16:54:56.2473934Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2022-09-27T16:54:56.2636584Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2022-09-27T16:54:56.2797533Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2022-09-27T16:54:56.2961280Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2022-09-27T16:54:56.3133278Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2022-09-27T16:54:56.3322660Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2022-09-27T16:54:56.3499171Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2022-09-27T16:54:56.3770827Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2022-09-27T16:54:56.3961230Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2022-09-27T16:54:56.4137648Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2022-09-27T16:54:56.4409890Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2022-09-27T16:54:56.4642906Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2022-09-27T16:54:56.5329347Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2022-09-27T16:54:56.5665128Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2022-09-27T16:54:56.5878353Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2022-09-27T16:54:56.6038351Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2022-09-27T16:54:56.6199045Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2022-09-27T16:54:57.8872626Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions PASSED    [ 75%]
2022-09-27T16:54:57.9133310Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2022-09-27T16:54:57.9287026Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2022-09-27T16:54:57.9445724Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2022-09-27T16:54:57.9596039Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2022-09-27T16:54:57.9751234Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2022-09-27T16:54:57.9908685Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device PASSED [ 85%]
2022-09-27T16:54:59.8098319Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules PASSED [ 87%]
2022-09-27T16:54:59.8357430Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2022-09-27T16:55:00.1545652Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2022-09-27T16:55:00.1717271Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] PASSED [ 92%]
2022-09-27T16:55:00.1898037Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] PASSED [ 94%]
2022-09-27T16:55:00.2174666Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] PASSED [ 96%]
2022-09-27T16:55:00.2336414Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device PASSED  [ 98%]
2022-09-27T16:55:00.2862468Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper PASSED  [100%]
2022-09-27T16:55:00.2863988Z 
2022-09-27T16:55:00.2864481Z ============================== 56 passed in 4.99s ==============================
2022-09-27T16:55:00.7243077Z Running distributed/pipeline/sync/test_transparency ... [2022-09-27 16:55:00.723785]
2022-09-27T16:55:00.7243791Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-09-27 16:55:00.723869]
2022-09-27T16:55:02.8216164Z ============================= test session starts ==============================
2022-09-27T16:55:02.8216751Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-09-27T16:55:02.8313095Z cachedir: .pytest_cache
2022-09-27T16:55:02.8313869Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-09-27T16:55:02.8314347Z torch: 1.13.0a0+git52424e2
2022-09-27T16:55:02.8314692Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-09-27T16:55:02.8315229Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, xdist-2.5.0, xdoctest-1.0.2
2022-09-27T16:55:02.8435027Z collecting ... 
2022-09-27T16:55:02.8435456Z collected 1 item                                                               
2022-09-27T16:55:02.8439711Z 
2022-09-27T16:55:02.9702138Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2022-09-27T16:55:02.9703188Z 
2022-09-27T16:55:02.9703660Z ============================== 1 passed in 0.15s ===============================
2022-09-27T16:55:03.2426652Z Running distributed/rpc/test_tensorpipe_agent ... [2022-09-27 16:55:03.242214]
2022-09-27T16:55:03.2427475Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-09-27 16:55:03.242294]
2022-09-27T16:55:05.0551739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_q5t5bxv
2022-09-27T16:55:05.0552750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_q5t5bxv/_remote_module_non_scriptable.py
2022-09-27T16:55:06.0359835Z 
2022-09-27T16:55:06.0360600Z real	68m16.914s
2022-09-27T16:55:06.0360955Z user	247m53.687s
2022-09-27T16:55:06.0361198Z sys	131m52.006s
2022-09-27T16:55:06.0361435Z + assert_git_not_dirty
2022-09-27T16:55:06.0361974Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-09-27T16:55:06.0362427Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]]
2022-09-27T16:55:06.0363823Z ++ git status --porcelain
2022-09-27T16:55:07.2509345Z + git_status=
2022-09-27T16:55:07.2509839Z + [[ -n '' ]]
2022-09-27T16:55:07.2510253Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-09-27T16:55:07.2510546Z + [[ 1 == 1 ]]
2022-09-27T16:55:07.2510875Z + echo 'Testing distributed C++ tests'
2022-09-27T16:55:07.2511187Z Testing distributed C++ tests
2022-09-27T16:55:07.2513072Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-09-27T16:55:07.2525435Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-09-27T16:55:07.2535947Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed
2022-09-27T16:55:07.2536453Z + mkdir -p test/test-reports/cpp-distributed/test_distributed
2022-09-27T16:55:07.2551038Z + /opt/conda/lib/python3.10/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml
2022-09-27T16:55:07.5859015Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-09-27T16:55:07.5859647Z [==========] Running 4 tests from 1 test suite.
2022-09-27T16:55:07.5860067Z [----------] Global test environment set-up.
2022-09-27T16:55:07.5860474Z [----------] 4 tests from FileStoreTest
2022-09-27T16:55:07.5861075Z [ RUN      ] FileStoreTest.testGetAndSet
2022-09-27T16:55:07.5864419Z [       OK ] FileStoreTest.testGetAndSet (0 ms)
2022-09-27T16:55:07.5864902Z [ RUN      ] FileStoreTest.testGetAndSetWithPrefix
2022-09-27T16:55:07.5869084Z [       OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms)
2022-09-27T16:55:07.5869545Z [ RUN      ] FileStoreTest.testStressStore
2022-09-27T16:55:07.6065917Z [       OK ] FileStoreTest.testStressStore (19 ms)
2022-09-27T16:55:07.6066414Z [ RUN      ] FileStoreTest.testStressStoreWithPrefix
2022-09-27T16:55:07.6264342Z [       OK ] FileStoreTest.testStressStoreWithPrefix (19 ms)
2022-09-27T16:55:07.6264841Z [----------] 4 tests from FileStoreTest (40 ms total)
2022-09-27T16:55:07.6265066Z 
2022-09-27T16:55:07.6265307Z [----------] Global test environment tear-down
2022-09-27T16:55:07.6267523Z [==========] 4 tests from 1 test suite ran. (40 ms total)
2022-09-27T16:55:07.6267885Z [  PASSED  ] 4 tests.
2022-09-27T16:55:07.6925082Z + /opt/conda/lib/python3.10/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml
2022-09-27T16:55:08.0266244Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-09-27T16:55:08.0266902Z [==========] Running 4 tests from 1 test suite.
2022-09-27T16:55:08.0267329Z [----------] Global test environment set-up.
2022-09-27T16:55:08.0267716Z [----------] 4 tests from HashStoreTest
2022-09-27T16:55:08.0268117Z [ RUN      ] HashStoreTest.testGetAndSet
2022-09-27T16:55:08.1270446Z [       OK ] HashStoreTest.testGetAndSet (100 ms)
2022-09-27T16:55:08.1270957Z [ RUN      ] HashStoreTest.testGetAndSetWithPrefix
2022-09-27T16:55:08.2275300Z [       OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms)
2022-09-27T16:55:08.2275785Z [ RUN      ] HashStoreTest.testStressStore
2022-09-27T16:55:08.2283284Z [       OK ] HashStoreTest.testStressStore (0 ms)
2022-09-27T16:55:08.2283758Z [ RUN      ] HashStoreTest.testStressStoreWithPrefix
2022-09-27T16:55:08.2289594Z [       OK ] HashStoreTest.testStressStoreWithPrefix (0 ms)
2022-09-27T16:55:08.2290101Z [----------] 4 tests from HashStoreTest (202 ms total)
2022-09-27T16:55:08.2290511Z 
2022-09-27T16:55:08.2290836Z [----------] Global test environment tear-down
2022-09-27T16:55:08.2292914Z [==========] 4 tests from 1 test suite ran. (202 ms total)
2022-09-27T16:55:08.2293292Z [  PASSED  ] 4 tests.
2022-09-27T16:55:08.2988317Z + /opt/conda/lib/python3.10/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml
2022-09-27T16:55:08.6294201Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-09-27T16:55:08.6294830Z [==========] Running 11 tests from 1 test suite.
2022-09-27T16:55:08.6295269Z [----------] Global test environment set-up.
2022-09-27T16:55:08.6295672Z [----------] 11 tests from TCPStoreTest
2022-09-27T16:55:08.6296043Z [ RUN      ] TCPStoreTest.testHelper
2022-09-27T16:55:09.6312361Z [       OK ] TCPStoreTest.testHelper (1001 ms)
2022-09-27T16:55:09.6312825Z [ RUN      ] TCPStoreTest.testHelperPrefix
2022-09-27T16:55:10.6148988Z [       OK ] TCPStoreTest.testHelperPrefix (983 ms)
2022-09-27T16:55:10.6149496Z [ RUN      ] TCPStoreTest.testWatchKeyCallback
2022-09-27T16:55:10.6293956Z [       OK ] TCPStoreTest.testWatchKeyCallback (14 ms)
2022-09-27T16:55:10.6294503Z [ RUN      ] TCPStoreTest.testWatchKeyCallbackWithPrefix
2022-09-27T16:55:10.6433624Z [       OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (13 ms)
2022-09-27T16:55:10.6434138Z [ RUN      ] TCPStoreTest.testKeyEmptyUpdate
2022-09-27T16:55:10.8487560Z [       OK ] TCPStoreTest.testKeyEmptyUpdate (205 ms)
2022-09-27T16:55:10.8488023Z [ RUN      ] TCPStoreTest.testKeyUpdate
2022-09-27T16:55:10.8494060Z [       OK ] TCPStoreTest.testKeyUpdate (0 ms)
2022-09-27T16:55:10.8494473Z [ RUN      ] TCPStoreTest.testKeyCreate
2022-09-27T16:55:10.8499509Z [       OK ] TCPStoreTest.testKeyCreate (0 ms)
2022-09-27T16:55:10.8499930Z [ RUN      ] TCPStoreTest.testKeyAdd
2022-09-27T16:55:10.8505118Z [       OK ] TCPStoreTest.testKeyAdd (0 ms)
2022-09-27T16:55:10.8505522Z [ RUN      ] TCPStoreTest.testKeyDelete
2022-09-27T16:55:11.0567347Z [       OK ] TCPStoreTest.testKeyDelete (206 ms)
2022-09-27T16:55:11.0567822Z [ RUN      ] TCPStoreTest.testCleanShutdown
2022-09-27T16:55:11.0575459Z [       OK ] TCPStoreTest.testCleanShutdown (0 ms)
2022-09-27T16:55:11.0575926Z [ RUN      ] TCPStoreTest.testMultiTenantStores
2022-09-27T16:55:11.0589367Z [       OK ] TCPStoreTest.testMultiTenantStores (1 ms)
2022-09-27T16:55:11.0589869Z [----------] 11 tests from TCPStoreTest (2429 ms total)
2022-09-27T16:55:11.0590105Z 
2022-09-27T16:55:11.0590344Z [----------] Global test environment tear-down
2022-09-27T16:55:11.0594464Z [==========] 11 tests from 1 test suite ran. (2429 ms total)
2022-09-27T16:55:11.0594844Z [  PASSED  ] 11 tests.
2022-09-27T16:55:11.1307995Z ++ command -v mpiexec
2022-09-27T16:55:11.1310646Z + MPIEXEC=/usr/bin/mpiexec
2022-09-27T16:55:11.1310998Z + [[ -n /usr/bin/mpiexec ]]
2022-09-27T16:55:11.1311288Z + [[ -z true ]]
2022-09-27T16:55:11.1311973Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml
2022-09-27T16:55:11.5675562Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-09-27T16:55:11.5676181Z [==========] Running 12 tests from 1 test suite.
2022-09-27T16:55:11.5676608Z [----------] Global test environment set-up.
2022-09-27T16:55:11.5677042Z [----------] 12 tests from ProcessGroupGlooTest
2022-09-27T16:55:11.5677504Z [ RUN      ] ProcessGroupGlooTest.testSIGSTOPException
2022-09-27T16:55:12.6116312Z [       OK ] ProcessGroupGlooTest.testSIGSTOPException (1043 ms)
2022-09-27T16:55:12.6116881Z [ RUN      ] ProcessGroupGlooTest.testSIGKILLException
2022-09-27T16:55:12.6353680Z [       OK ] ProcessGroupGlooTest.testSIGKILLException (23 ms)
2022-09-27T16:55:12.6354192Z [ RUN      ] ProcessGroupGlooTest.testAllReduceCPU
2022-09-27T16:55:12.9436897Z [       OK ] ProcessGroupGlooTest.testAllReduceCPU (308 ms)
2022-09-27T16:55:12.9437698Z [ RUN      ] ProcessGroupGlooTest.testBroadcastCPU
2022-09-27T16:55:12.9860979Z [       OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms)
2022-09-27T16:55:12.9861485Z [ RUN      ] ProcessGroupGlooTest.testAllToAllCPU
2022-09-27T16:55:13.1296191Z [       OK ] ProcessGroupGlooTest.testAllToAllCPU (143 ms)
2022-09-27T16:55:13.1296686Z [ RUN      ] ProcessGroupGlooTest.testBarrier
2022-09-27T16:55:13.1813519Z [       OK ] ProcessGroupGlooTest.testBarrier (51 ms)
2022-09-27T16:55:13.1814015Z [ RUN      ] ProcessGroupGlooTest.testMonitoredBarrier
2022-09-27T16:55:14.2018055Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms
2022-09-27T16:55:14.2220188Z [       OK ] ProcessGroupGlooTest.testMonitoredBarrier (1040 ms)
2022-09-27T16:55:14.2220748Z [ RUN      ] ProcessGroupGlooTest.testSequenceNumInit
2022-09-27T16:55:14.3643679Z [       OK ] ProcessGroupGlooTest.testSequenceNumInit (142 ms)
2022-09-27T16:55:14.3644196Z [ RUN      ] ProcessGroupGlooTest.testSend
2022-09-27T16:55:14.4160962Z [       OK ] ProcessGroupGlooTest.testSend (51 ms)
2022-09-27T16:55:14.4161671Z [ RUN      ] ProcessGroupGlooTest.testRecv
2022-09-27T16:55:14.4580305Z [       OK ] ProcessGroupGlooTest.testRecv (41 ms)
2022-09-27T16:55:14.4580801Z [ RUN      ] ProcessGroupGlooTest.testStoreSetGet
2022-09-27T16:55:14.4993398Z [       OK ] ProcessGroupGlooTest.testStoreSetGet (41 ms)
2022-09-27T16:55:14.4993881Z [ RUN      ] ProcessGroupGlooTest.testWaitDelay
2022-09-27T16:55:14.6416136Z [       OK ] ProcessGroupGlooTest.testWaitDelay (142 ms)
2022-09-27T16:55:14.6416669Z [----------] 12 tests from ProcessGroupGlooTest (3074 ms total)
2022-09-27T16:55:14.6416919Z 
2022-09-27T16:55:14.6417153Z [----------] Global test environment tear-down
2022-09-27T16:55:14.6420062Z [==========] 12 tests from 1 test suite ran. (3074 ms total)
2022-09-27T16:55:14.6420447Z [  PASSED  ] 12 tests.
2022-09-27T16:55:14.7368802Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml
2022-09-27T16:55:15.1691786Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-09-27T16:55:15.1692425Z [==========] Running 11 tests from 1 test suite.
2022-09-27T16:55:15.1693010Z [----------] Global test environment set-up.
2022-09-27T16:55:15.1693448Z [----------] 11 tests from ProcessGroupNCCLTest
2022-09-27T16:55:15.1693898Z [ RUN      ] ProcessGroupNCCLTest.testAllreduce
2022-09-27T16:55:19.5642480Z [       OK ] ProcessGroupNCCLTest.testAllreduce (4394 ms)
2022-09-27T16:55:19.5643036Z [ RUN      ] ProcessGroupNCCLTest.testBroadcast
2022-09-27T16:55:23.0659692Z [       OK ] ProcessGroupNCCLTest.testBroadcast (3501 ms)
2022-09-27T16:55:23.0660215Z [ RUN      ] ProcessGroupNCCLTest.testReduce
2022-09-27T16:55:26.5673344Z [       OK ] ProcessGroupNCCLTest.testReduce (3501 ms)
2022-09-27T16:55:26.5673880Z [ RUN      ] ProcessGroupNCCLTest.testAllgather
2022-09-27T16:55:28.3500963Z [       OK ] ProcessGroupNCCLTest.testAllgather (1782 ms)
2022-09-27T16:55:28.3501525Z [ RUN      ] ProcessGroupNCCLTest.testAllgatherBase
2022-09-27T16:55:30.1209254Z [       OK ] ProcessGroupNCCLTest.testAllgatherBase (1770 ms)
2022-09-27T16:55:30.1209819Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatter
2022-09-27T16:55:31.9052160Z [       OK ] ProcessGroupNCCLTest.testReduceScatter (1784 ms)
2022-09-27T16:55:31.9052727Z [ RUN      ] ProcessGroupNCCLTest.testSequenceNumInit
2022-09-27T16:55:32.0202821Z [       OK ] ProcessGroupNCCLTest.testSequenceNumInit (114 ms)
2022-09-27T16:55:32.0203735Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout
2022-09-27T16:55:35.0221801Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3001 ms)
2022-09-27T16:55:35.0222595Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException
2022-09-27T16:55:38.0233935Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms)
2022-09-27T16:55:38.0234621Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatterBase
2022-09-27T16:55:40.1609165Z [       OK ] ProcessGroupNCCLTest.testReduceScatterBase (2137 ms)
2022-09-27T16:55:40.1609704Z [ RUN      ] ProcessGroupNCCLTest.testBackendName
2022-09-27T16:55:40.2035417Z [       OK ] ProcessGroupNCCLTest.testBackendName (42 ms)
2022-09-27T16:55:40.2035989Z [----------] 11 tests from ProcessGroupNCCLTest (25034 ms total)
2022-09-27T16:55:40.2036243Z 
2022-09-27T16:55:40.2038461Z [----------] Global test environment tear-down
2022-09-27T16:55:40.2039046Z [==========] 11 tests from 1 test suite ran. (25034 ms total)
2022-09-27T16:55:40.2039421Z [  PASSED  ] 11 tests.
2022-09-27T16:55:40.4263454Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml
2022-09-27T16:55:40.8616470Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-09-27T16:55:40.8617301Z [==========] Running 3 tests from 1 test suite.
2022-09-27T16:55:40.8617849Z [----------] Global test environment set-up.
2022-09-27T16:55:40.8618307Z [----------] 3 tests from ProcessGroupNCCLErrorsTest
2022-09-27T16:55:40.8618836Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking
2022-09-27T16:55:43.1818355Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (2320 ms)
2022-09-27T16:55:43.1819106Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking
2022-09-27T16:55:46.2901810Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3108 ms)
2022-09-27T16:55:46.2902542Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking
2022-09-27T16:55:46.3921140Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (101 ms)
2022-09-27T16:55:46.3921803Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (5530 ms total)
2022-09-27T16:55:46.3922263Z 
2022-09-27T16:55:46.3922584Z [----------] Global test environment tear-down
2022-09-27T16:55:46.3923029Z [==========] 3 tests from 1 test suite ran. (5530 ms total)
2022-09-27T16:55:46.3923388Z [  PASSED  ] 3 tests.
2022-09-27T16:55:46.6088577Z + [[ 1 == 1 ]]
2022-09-27T16:55:46.6088922Z + test_rpc
2022-09-27T16:55:46.6089407Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-09-27T16:55:46.6089789Z + echo 'Testing RPC C++ tests'
2022-09-27T16:55:46.6090086Z Testing RPC C++ tests
2022-09-27T16:55:46.6091728Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-09-27T16:55:46.6108082Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-09-27T16:55:46.6120416Z + ln -sf '/opt/conda/lib/python3.10/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.10/site-packages/torch/bin
2022-09-27T16:55:46.6130406Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-09-27T16:55:46.6131103Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-09-27T16:55:46.6143629Z + /opt/conda/lib/python3.10/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-09-27T16:55:47.0698506Z [==========] Running 8 tests from 3 test suites.
2022-09-27T16:55:47.0698989Z [----------] Global test environment set-up.
2022-09-27T16:55:47.0699437Z [----------] 4 tests from WireSerialize
2022-09-27T16:55:47.0699908Z [ RUN      ] WireSerialize.Base
2022-09-27T16:55:47.0712382Z [       OK ] WireSerialize.Base (1 ms)
2022-09-27T16:55:47.0713086Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-09-27T16:55:47.0814847Z [       OK ] WireSerialize.RecopySparseTensors (10 ms)
2022-09-27T16:55:47.0815462Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-09-27T16:55:47.0907735Z [       OK ] WireSerialize.CloneSparseTensors (9 ms)
2022-09-27T16:55:47.0908763Z [ RUN      ] WireSerialize.Errors
2022-09-27T16:55:47.0935679Z [       OK ] WireSerialize.Errors (2 ms)
2022-09-27T16:55:47.0936331Z [----------] 4 tests from WireSerialize (23 ms total)
2022-09-27T16:55:47.0936604Z 
2022-09-27T16:55:47.0936842Z [----------] 1 test from TestE2ETensorPipe
2022-09-27T16:55:47.0937282Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-09-27T16:55:47.8155929Z [W tensorpipe_agent.cpp:726] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-09-27T16:55:47.8178012Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (724 ms)
2022-09-27T16:55:47.8178567Z [----------] 1 test from TestE2ETensorPipe (724 ms total)
2022-09-27T16:55:47.8178815Z 
2022-09-27T16:55:47.8179055Z [----------] 3 tests from TensorpipeSerialize
2022-09-27T16:55:47.8179473Z [ RUN      ] TensorpipeSerialize.Base
2022-09-27T16:55:47.8179893Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-09-27T16:55:47.8180365Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-09-27T16:55:47.8277775Z [       OK ] TensorpipeSerialize.RecopySparseTensors (9 ms)
2022-09-27T16:55:47.8278771Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-09-27T16:55:47.8279703Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-09-27T16:55:47.8281160Z [----------] 3 tests from TensorpipeSerialize (9 ms total)
2022-09-27T16:55:47.8281615Z 
2022-09-27T16:55:47.8282049Z [----------] Global test environment tear-down
2022-09-27T16:55:47.8283111Z [==========] 8 tests from 3 test suites ran. (758 ms total)
2022-09-27T16:55:47.8283818Z [  PASSED  ] 8 tests.
2022-09-27T16:55:47.8284150Z 
2022-09-27T16:55:47.8284469Z   YOU HAVE 1 DISABLED TEST
2022-09-27T16:55:47.8284803Z 
2022-09-27T16:55:47.9327419Z 
2022-09-27T16:55:47.9364236Z Prepare all required actions
2022-09-27T16:55:47.9364602Z Getting action download info
2022-09-27T16:55:48.3541852Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-09-27T16:55:48.3542160Z with:
2022-09-27T16:55:48.3542606Z   github-token: ***
2022-09-27T16:55:48.3542853Z env:
2022-09-27T16:55:48.3543099Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:48.3543357Z   GPU_FLAG: --gpus all
2022-09-27T16:55:48.3543617Z ##[endgroup]
2022-09-27T16:55:48.3578512Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-09-27T16:55:48.3578819Z with:
2022-09-27T16:55:48.3579049Z   shell: bash
2022-09-27T16:55:48.3579298Z   timeout_minutes: 10
2022-09-27T16:55:48.3579534Z   max_attempts: 5
2022-09-27T16:55:48.3579791Z   retry_wait_seconds: 30
2022-09-27T16:55:48.3580349Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-09-27T16:55:48.3580849Z   polling_interval_seconds: 1
2022-09-27T16:55:48.3581125Z   warning_on_retry: true
2022-09-27T16:55:48.3581393Z   continue_on_error: false
2022-09-27T16:55:48.3581622Z env:
2022-09-27T16:55:48.3581864Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:48.3582147Z   GPU_FLAG: --gpus all
2022-09-27T16:55:48.3582529Z   GITHUB_TOKEN: ***
2022-09-27T16:55:48.3582776Z ##[endgroup]
2022-09-27T16:55:48.4148980Z 
2022-09-27T16:55:48.4218961Z + python3 -m pip install requests==2.26.0
2022-09-27T16:55:48.7067715Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:55:48.8472414Z Collecting requests==2.26.0
2022-09-27T16:55:48.8658868Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-09-27T16:55:48.9919711Z Collecting urllib3<1.27,>=1.21.1
2022-09-27T16:55:48.9964455Z   Downloading urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
2022-09-27T16:55:49.0933486Z Collecting certifi>=2017.4.17
2022-09-27T16:55:49.0985873Z   Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
2022-09-27T16:55:49.2044972Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-09-27T16:55:49.2088856Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-09-27T16:55:49.2541110Z Collecting idna<4,>=2.5; python_version >= "3"
2022-09-27T16:55:49.2585397Z   Downloading idna-3.4-py3-none-any.whl (61 kB)
2022-09-27T16:55:49.3510352Z Installing collected packages: urllib3, certifi, charset-normalizer, idna, requests
2022-09-27T16:55:49.4836359Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:55:49.4837044Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:55:49.6172447Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.26.0 urllib3-1.26.12
2022-09-27T16:55:49.6643970Z ++ python3 .github/scripts/get_workflow_job_id.py 3133193930 i-037610acd6d6b81a9
2022-09-27T16:55:54.9700453Z + GHA_WORKFLOW_JOB_ID=8576432118
2022-09-27T16:55:54.9701500Z + echo '::set-output name=job-id::8576432118'
2022-09-27T16:55:55.4280937Z Command completed after 1 attempt(s).
2022-09-27T16:55:55.4281732Z 
2022-09-27T16:55:55.4423393Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-09-27T16:55:55.4423721Z kill "$MONITOR_SCRIPT_PID"
2022-09-27T16:55:55.4436865Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:55:55.4437170Z env:
2022-09-27T16:55:55.4437399Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:55.4437669Z   GPU_FLAG: --gpus all
2022-09-27T16:55:55.4437938Z   MONITOR_SCRIPT_PID: 56727
2022-09-27T16:55:55.4438181Z ##[endgroup]
2022-09-27T16:55:55.4532827Z Prepare all required actions
2022-09-27T16:55:55.4533212Z Getting action download info
2022-09-27T16:55:55.8208426Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-09-27T16:55:56.6616729Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-09-27T16:55:56.6617044Z with:
2022-09-27T16:55:56.6617385Z   file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118
2022-09-27T16:55:56.6617733Z env:
2022-09-27T16:55:56.6617974Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:56.6618227Z   GPU_FLAG: --gpus all
2022-09-27T16:55:56.6618473Z ##[endgroup]
2022-09-27T16:55:56.6650405Z ##[group]Run # Remove any previous test jsons if they exist
2022-09-27T16:55:56.6650790Z # Remove any previous test jsons if they exist
2022-09-27T16:55:56.6651111Z rm -f test-jsons-*.zip
2022-09-27T16:55:56.6651435Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-09-27T16:55:56.6663827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:55:56.6664132Z env:
2022-09-27T16:55:56.6664361Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:56.6664632Z   GPU_FLAG: --gpus all
2022-09-27T16:55:56.6665008Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118
2022-09-27T16:55:56.6665349Z ##[endgroup]
2022-09-27T16:55:56.6843052Z   adding: test/allowlist_for_publicAPI.json (deflated 80%)
2022-09-27T16:55:56.6876881Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-09-27T16:55:56.6884587Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-09-27T16:55:56.6886082Z   adding: test/.pytorch-slow-tests.json (deflated 75%)
2022-09-27T16:55:56.6892667Z   adding: test/.pytorch-disabled-tests.json (deflated 85%)
2022-09-27T16:55:56.6933627Z ##[group]Run # Remove any previous test reports if they exist
2022-09-27T16:55:56.6934022Z # Remove any previous test reports if they exist
2022-09-27T16:55:56.6934363Z rm -f test-reports-*.zip
2022-09-27T16:55:56.6934693Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-09-27T16:55:56.6946846Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:55:56.6947146Z env:
2022-09-27T16:55:56.6947387Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:56.6947638Z   GPU_FLAG: --gpus all
2022-09-27T16:55:56.6948146Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118
2022-09-27T16:55:56.6948500Z ##[endgroup]
2022-09-27T16:55:56.7168516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154655.xml (deflated 41%)
2022-09-27T16:55:56.7169346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154706.xml (deflated 41%)
2022-09-27T16:55:56.7170142Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154708.xml (deflated 41%)
2022-09-27T16:55:56.7170946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154715.xml (deflated 64%)
2022-09-27T16:55:56.7171716Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154837.xml (deflated 41%)
2022-09-27T16:55:56.7172497Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154845.xml (deflated 41%)
2022-09-27T16:55:56.7173283Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154852.xml (deflated 40%)
2022-09-27T16:55:56.7174066Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154901.xml (deflated 40%)
2022-09-27T16:55:56.7174828Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154910.xml (deflated 40%)
2022-09-27T16:55:56.7175743Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154919.xml (deflated 39%)
2022-09-27T16:55:56.7176564Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154928.xml (deflated 39%)
2022-09-27T16:55:56.7177336Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154930.xml (deflated 40%)
2022-09-27T16:55:56.7178096Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154939.xml (deflated 40%)
2022-09-27T16:55:56.7178868Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154948.xml (deflated 42%)
2022-09-27T16:55:56.7179637Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154952.xml (deflated 41%)
2022-09-27T16:55:56.7180412Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154959.xml (deflated 41%)
2022-09-27T16:55:56.7181163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155008.xml (deflated 42%)
2022-09-27T16:55:56.7181949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155017.xml (deflated 42%)
2022-09-27T16:55:56.7182725Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155020.xml (deflated 46%)
2022-09-27T16:55:56.7183496Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155027.xml (deflated 47%)
2022-09-27T16:55:56.7184247Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155034.xml (deflated 48%)
2022-09-27T16:55:56.7185016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155042.xml (deflated 45%)
2022-09-27T16:55:56.7185781Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155049.xml (deflated 40%)
2022-09-27T16:55:56.7186549Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155056.xml (deflated 40%)
2022-09-27T16:55:56.7187427Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155103.xml (deflated 40%)
2022-09-27T16:55:56.7188178Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155110.xml (deflated 42%)
2022-09-27T16:55:56.7188954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155117.xml (deflated 41%)
2022-09-27T16:55:56.7189736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155124.xml (deflated 41%)
2022-09-27T16:55:56.7190508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155132.xml (deflated 40%)
2022-09-27T16:55:56.7191256Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155139.xml (deflated 41%)
2022-09-27T16:55:56.7212838Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155141.xml (deflated 41%)
2022-09-27T16:55:56.7213671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155144.xml (deflated 40%)
2022-09-27T16:55:56.7214461Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155151.xml (deflated 42%)
2022-09-27T16:55:56.7215228Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155158.xml (deflated 42%)
2022-09-27T16:55:56.7216159Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155200.xml (deflated 42%)
2022-09-27T16:55:56.7216984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155203.xml (deflated 40%)
2022-09-27T16:55:56.7217777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155210.xml (deflated 40%)
2022-09-27T16:55:56.7218555Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155217.xml (deflated 43%)
2022-09-27T16:55:56.7219329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155219.xml (deflated 40%)
2022-09-27T16:55:56.7220094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155227.xml (deflated 40%)
2022-09-27T16:55:56.7220882Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155234.xml (deflated 40%)
2022-09-27T16:55:56.7221660Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155241.xml (deflated 40%)
2022-09-27T16:55:56.7222492Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155248.xml (deflated 41%)
2022-09-27T16:55:56.7223265Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155255.xml (deflated 41%)
2022-09-27T16:55:56.7224028Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155303.xml (deflated 41%)
2022-09-27T16:55:56.7224802Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155310.xml (deflated 42%)
2022-09-27T16:55:56.7225577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155317.xml (deflated 40%)
2022-09-27T16:55:56.7226356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155324.xml (deflated 40%)
2022-09-27T16:55:56.7227109Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155331.xml (deflated 40%)
2022-09-27T16:55:56.7228007Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155338.xml (deflated 40%)
2022-09-27T16:55:56.7228791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155346.xml (deflated 40%)
2022-09-27T16:55:56.7229562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155353.xml (deflated 40%)
2022-09-27T16:55:56.7230322Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155400.xml (deflated 40%)
2022-09-27T16:55:56.7231099Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155407.xml (deflated 40%)
2022-09-27T16:55:56.7231866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155415.xml (deflated 40%)
2022-09-27T16:55:56.7232639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155422.xml (deflated 40%)
2022-09-27T16:55:56.7233392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155429.xml (deflated 42%)
2022-09-27T16:55:56.7234158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155436.xml (deflated 42%)
2022-09-27T16:55:56.7234989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155443.xml (deflated 41%)
2022-09-27T16:55:56.7235768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155451.xml (deflated 42%)
2022-09-27T16:55:56.7236525Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155458.xml (deflated 40%)
2022-09-27T16:55:56.7237292Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155505.xml (deflated 40%)
2022-09-27T16:55:56.7238059Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155512.xml (deflated 41%)
2022-09-27T16:55:56.7238829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155521.xml (deflated 41%)
2022-09-27T16:55:56.7239581Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155530.xml (deflated 40%)
2022-09-27T16:55:56.7240784Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155537.xml (deflated 40%)
2022-09-27T16:55:56.7241589Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155545.xml (deflated 40%)
2022-09-27T16:55:56.7242369Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155552.xml (deflated 41%)
2022-09-27T16:55:56.7243124Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155559.xml (deflated 40%)
2022-09-27T16:55:56.7243898Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155606.xml (deflated 40%)
2022-09-27T16:55:56.7244665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155618.xml (deflated 40%)
2022-09-27T16:55:56.7245437Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155628.xml (deflated 41%)
2022-09-27T16:55:56.7246194Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155639.xml (deflated 42%)
2022-09-27T16:55:56.7247087Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155642.xml (deflated 42%)
2022-09-27T16:55:56.7247854Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155644.xml (deflated 42%)
2022-09-27T16:55:56.7248630Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155647.xml (deflated 41%)
2022-09-27T16:55:56.7249378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155649.xml (deflated 42%)
2022-09-27T16:55:56.7250163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155652.xml (deflated 41%)
2022-09-27T16:55:56.7250936Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155654.xml (deflated 42%)
2022-09-27T16:55:56.7251705Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155657.xml (deflated 42%)
2022-09-27T16:55:56.7252462Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155659.xml (deflated 42%)
2022-09-27T16:55:56.7253231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155702.xml (deflated 42%)
2022-09-27T16:55:56.7253996Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155704.xml (deflated 42%)
2022-09-27T16:55:56.7254833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155707.xml (deflated 42%)
2022-09-27T16:55:56.7255619Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155709.xml (deflated 42%)
2022-09-27T16:55:56.7256379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155712.xml (deflated 42%)
2022-09-27T16:55:56.7257154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155714.xml (deflated 42%)
2022-09-27T16:55:56.7257931Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155717.xml (deflated 42%)
2022-09-27T16:55:56.7258705Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155719.xml (deflated 42%)
2022-09-27T16:55:56.7259460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155722.xml (deflated 42%)
2022-09-27T16:55:56.7260281Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155724.xml (deflated 42%)
2022-09-27T16:55:56.7261052Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155727.xml (deflated 42%)
2022-09-27T16:55:56.7261826Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155729.xml (deflated 42%)
2022-09-27T16:55:56.7262583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155732.xml (deflated 42%)
2022-09-27T16:55:56.7263359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155734.xml (deflated 42%)
2022-09-27T16:55:56.7264124Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155737.xml (deflated 42%)
2022-09-27T16:55:56.7264893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155739.xml (deflated 41%)
2022-09-27T16:55:56.7265644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155749.xml (deflated 41%)
2022-09-27T16:55:56.7266482Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155756.xml (deflated 42%)
2022-09-27T16:55:56.7267252Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155759.xml (deflated 40%)
2022-09-27T16:55:56.7268029Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155807.xml (deflated 40%)
2022-09-27T16:55:56.7268780Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155816.xml (deflated 40%)
2022-09-27T16:55:56.7269563Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155824.xml (deflated 42%)
2022-09-27T16:55:56.7270334Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155831.xml (deflated 42%)
2022-09-27T16:55:56.7271112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155838.xml (deflated 42%)
2022-09-27T16:55:56.7271869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155846.xml (deflated 40%)
2022-09-27T16:55:56.7272643Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155854.xml (deflated 42%)
2022-09-27T16:55:56.7273407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155856.xml (deflated 42%)
2022-09-27T16:55:56.7274226Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155903.xml (deflated 41%)
2022-09-27T16:55:56.7274994Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155911.xml (deflated 40%)
2022-09-27T16:55:56.7275763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155918.xml (deflated 41%)
2022-09-27T16:55:56.7276534Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155920.xml (deflated 41%)
2022-09-27T16:55:56.7277308Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155923.xml (deflated 40%)
2022-09-27T16:55:56.7278142Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155925.xml (deflated 41%)
2022-09-27T16:55:56.7278899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155928.xml (deflated 41%)
2022-09-27T16:55:56.7279671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155931.xml (deflated 40%)
2022-09-27T16:55:56.7280787Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155933.xml (deflated 41%)
2022-09-27T16:55:56.7281589Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155936.xml (deflated 41%)
2022-09-27T16:55:56.7282344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155938.xml (deflated 40%)
2022-09-27T16:55:56.7283120Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155945.xml (deflated 42%)
2022-09-27T16:55:56.7283892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155950.xml (deflated 40%)
2022-09-27T16:55:56.7284660Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155957.xml (deflated 42%)
2022-09-27T16:55:56.7285410Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160004.xml (deflated 40%)
2022-09-27T16:55:56.7286284Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160013.xml (deflated 42%)
2022-09-27T16:55:56.7287052Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160018.xml (deflated 42%)
2022-09-27T16:55:56.7287825Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160022.xml (deflated 42%)
2022-09-27T16:55:56.7288595Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160026.xml (deflated 40%)
2022-09-27T16:55:56.7289344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160035.xml (deflated 40%)
2022-09-27T16:55:56.7290108Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160044.xml (deflated 40%)
2022-09-27T16:55:56.7290874Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160053.xml (deflated 40%)
2022-09-27T16:55:56.7291633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160103.xml (deflated 40%)
2022-09-27T16:55:56.7292381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160112.xml (deflated 40%)
2022-09-27T16:55:56.7293197Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160120.xml (deflated 41%)
2022-09-27T16:55:56.7293968Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160130.xml (deflated 40%)
2022-09-27T16:55:56.7294734Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160139.xml (deflated 40%)
2022-09-27T16:55:56.7295488Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160148.xml (deflated 40%)
2022-09-27T16:55:56.7296244Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160155.xml (deflated 41%)
2022-09-27T16:55:56.7297009Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160205.xml (deflated 41%)
2022-09-27T16:55:56.7297778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160214.xml (deflated 40%)
2022-09-27T16:55:56.7298528Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160223.xml (deflated 40%)
2022-09-27T16:55:56.7299292Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160232.xml (deflated 40%)
2022-09-27T16:55:56.7300061Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160242.xml (deflated 40%)
2022-09-27T16:55:56.7300818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160252.xml (deflated 40%)
2022-09-27T16:55:56.7301561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160302.xml (deflated 40%)
2022-09-27T16:55:56.7302329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160311.xml (deflated 41%)
2022-09-27T16:55:56.7303094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160320.xml (deflated 41%)
2022-09-27T16:55:56.7303857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160330.xml (deflated 41%)
2022-09-27T16:55:56.7304677Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160350.xml (deflated 40%)
2022-09-27T16:55:56.7305443Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160359.xml (deflated 40%)
2022-09-27T16:55:56.7306210Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160409.xml (deflated 40%)
2022-09-27T16:55:56.7306972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160418.xml (deflated 40%)
2022-09-27T16:55:56.7307723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160427.xml (deflated 40%)
2022-09-27T16:55:56.7308493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160437.xml (deflated 40%)
2022-09-27T16:55:56.7309258Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160446.xml (deflated 41%)
2022-09-27T16:55:56.7310032Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160456.xml (deflated 41%)
2022-09-27T16:55:56.7310779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160505.xml (deflated 41%)
2022-09-27T16:55:56.7311549Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160514.xml (deflated 41%)
2022-09-27T16:55:56.7312366Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160524.xml (deflated 40%)
2022-09-27T16:55:56.7313143Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160533.xml (deflated 40%)
2022-09-27T16:55:56.7314036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160543.xml (deflated 40%)
2022-09-27T16:55:56.7314820Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160550.xml (deflated 40%)
2022-09-27T16:55:56.7315588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160559.xml (deflated 40%)
2022-09-27T16:55:56.7316353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160608.xml (deflated 40%)
2022-09-27T16:55:56.7317108Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160627.xml (deflated 41%)
2022-09-27T16:55:56.7317875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160636.xml (deflated 41%)
2022-09-27T16:55:56.7318633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160646.xml (deflated 41%)
2022-09-27T16:55:56.7319402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160655.xml (deflated 40%)
2022-09-27T16:55:56.7320413Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160705.xml (deflated 40%)
2022-09-27T16:55:56.7321374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160714.xml (deflated 42%)
2022-09-27T16:55:56.7322157Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160724.xml (deflated 41%)
2022-09-27T16:55:56.7322923Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160734.xml (deflated 40%)
2022-09-27T16:55:56.7323676Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160743.xml (deflated 40%)
2022-09-27T16:55:56.7324559Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160752.xml (deflated 40%)
2022-09-27T16:55:56.7325330Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160806.xml (deflated 41%)
2022-09-27T16:55:56.7326095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160815.xml (deflated 41%)
2022-09-27T16:55:56.7326848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160825.xml (deflated 41%)
2022-09-27T16:55:56.7327617Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160834.xml (deflated 40%)
2022-09-27T16:55:56.7328384Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160843.xml (deflated 40%)
2022-09-27T16:55:56.7329154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160852.xml (deflated 42%)
2022-09-27T16:55:56.7329917Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160856.xml (deflated 42%)
2022-09-27T16:55:56.7330669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160901.xml (deflated 41%)
2022-09-27T16:55:56.7331432Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160920.xml (deflated 41%)
2022-09-27T16:55:56.7332272Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160928.xml (deflated 41%)
2022-09-27T16:55:56.7333056Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160935.xml (deflated 40%)
2022-09-27T16:55:56.7333808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160946.xml (deflated 41%)
2022-09-27T16:55:56.7334579Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160955.xml (deflated 40%)
2022-09-27T16:55:56.7335339Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161003.xml (deflated 40%)
2022-09-27T16:55:56.7336111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161010.xml (deflated 40%)
2022-09-27T16:55:56.7336866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161017.xml (deflated 41%)
2022-09-27T16:55:56.7337633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161019.xml (deflated 40%)
2022-09-27T16:55:56.7338407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161027.xml (deflated 42%)
2022-09-27T16:55:56.7339177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161034.xml (deflated 40%)
2022-09-27T16:55:56.7339927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161041.xml (deflated 42%)
2022-09-27T16:55:56.7340693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161045.xml (deflated 40%)
2022-09-27T16:55:56.7341468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161052.xml (deflated 40%)
2022-09-27T16:55:56.7342238Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161059.xml (deflated 40%)
2022-09-27T16:55:56.7342986Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161106.xml (deflated 40%)
2022-09-27T16:55:56.7343828Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161117.xml (deflated 40%)
2022-09-27T16:55:56.7344591Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161131.xml (deflated 40%)
2022-09-27T16:55:56.7345363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161140.xml (deflated 40%)
2022-09-27T16:55:56.7346114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161147.xml (deflated 40%)
2022-09-27T16:55:56.7346886Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161155.xml (deflated 40%)
2022-09-27T16:55:56.7347649Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161202.xml (deflated 41%)
2022-09-27T16:55:56.7348417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161209.xml (deflated 41%)
2022-09-27T16:55:56.7349169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161235.xml (deflated 40%)
2022-09-27T16:55:56.7349933Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161301.xml (deflated 42%)
2022-09-27T16:55:56.7350761Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161308.xml (deflated 40%)
2022-09-27T16:55:56.7351546Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161317.xml (deflated 40%)
2022-09-27T16:55:56.7352296Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161324.xml (deflated 40%)
2022-09-27T16:55:56.7353067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161331.xml (deflated 41%)
2022-09-27T16:55:56.7353833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161337.xml (deflated 41%)
2022-09-27T16:55:56.7354601Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161340.xml (deflated 42%)
2022-09-27T16:55:56.7355354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161342.xml (deflated 42%)
2022-09-27T16:55:56.7356127Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161345.xml (deflated 41%)
2022-09-27T16:55:56.7356894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161347.xml (deflated 42%)
2022-09-27T16:55:56.7357661Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161350.xml (deflated 41%)
2022-09-27T16:55:56.7358411Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161352.xml (deflated 41%)
2022-09-27T16:55:56.7359176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161354.xml (deflated 41%)
2022-09-27T16:55:56.7359982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161357.xml (deflated 40%)
2022-09-27T16:55:56.7361082Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161404.xml (deflated 40%)
2022-09-27T16:55:56.7361843Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161411.xml (deflated 41%)
2022-09-27T16:55:56.7362721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161413.xml (deflated 42%)
2022-09-27T16:55:56.7363593Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161416.xml (deflated 42%)
2022-09-27T16:55:56.7364341Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161420.xml (deflated 40%)
2022-09-27T16:55:56.7365110Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161429.xml (deflated 40%)
2022-09-27T16:55:56.7365887Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161438.xml (deflated 40%)
2022-09-27T16:55:56.7366651Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161448.xml (deflated 40%)
2022-09-27T16:55:56.7367400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161457.xml (deflated 39%)
2022-09-27T16:55:56.7368172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161506.xml (deflated 40%)
2022-09-27T16:55:56.7368935Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161512.xml (deflated 40%)
2022-09-27T16:55:56.7369699Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161519.xml (deflated 39%)
2022-09-27T16:55:56.7370522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161528.xml (deflated 40%)
2022-09-27T16:55:56.7371309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161535.xml (deflated 40%)
2022-09-27T16:55:56.7372079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161542.xml (deflated 40%)
2022-09-27T16:55:56.7372849Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161549.xml (deflated 40%)
2022-09-27T16:55:56.7373603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161556.xml (deflated 42%)
2022-09-27T16:55:56.7374371Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161603.xml (deflated 42%)
2022-09-27T16:55:56.7375138Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161610.xml (deflated 42%)
2022-09-27T16:55:56.7375913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161617.xml (deflated 43%)
2022-09-27T16:55:56.7376665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161623.xml (deflated 40%)
2022-09-27T16:55:56.7377436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161630.xml (deflated 41%)
2022-09-27T16:55:56.7378203Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161637.xml (deflated 42%)
2022-09-27T16:55:56.7378969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161639.xml (deflated 41%)
2022-09-27T16:55:56.7379719Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161645.xml (deflated 43%)
2022-09-27T16:55:56.7380498Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161648.xml (deflated 40%)
2022-09-27T16:55:56.7381261Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161654.xml (deflated 42%)
2022-09-27T16:55:56.7382109Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161657.xml (deflated 42%)
2022-09-27T16:55:56.7382857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161659.xml (deflated 41%)
2022-09-27T16:55:56.7383621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161706.xml (deflated 41%)
2022-09-27T16:55:56.7384389Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161712.xml (deflated 41%)
2022-09-27T16:55:56.7385158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161719.xml (deflated 40%)
2022-09-27T16:55:56.7385907Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161726.xml (deflated 41%)
2022-09-27T16:55:56.7386680Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161728.xml (deflated 41%)
2022-09-27T16:55:56.7387447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161730.xml (deflated 40%)
2022-09-27T16:55:56.7388212Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161737.xml (deflated 42%)
2022-09-27T16:55:56.7388964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161744.xml (deflated 40%)
2022-09-27T16:55:56.7389790Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161750.xml (deflated 40%)
2022-09-27T16:55:56.7390572Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161757.xml (deflated 40%)
2022-09-27T16:55:56.7391343Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161804.xml (deflated 40%)
2022-09-27T16:55:56.7392094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161810.xml (deflated 40%)
2022-09-27T16:55:56.7392864Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161817.xml (deflated 41%)
2022-09-27T16:55:56.7393630Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161824.xml (deflated 41%)
2022-09-27T16:55:56.7394402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161826.xml (deflated 40%)
2022-09-27T16:55:56.7395153Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161828.xml (deflated 40%)
2022-09-27T16:55:56.7395916Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161831.xml (deflated 40%)
2022-09-27T16:55:56.7396687Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161837.xml (deflated 41%)
2022-09-27T16:55:56.7397450Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161844.xml (deflated 40%)
2022-09-27T16:55:56.7398198Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161851.xml (deflated 40%)
2022-09-27T16:55:56.7398975Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161857.xml (deflated 40%)
2022-09-27T16:55:56.7399736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161904.xml (deflated 40%)
2022-09-27T16:55:56.7400845Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161912.xml (deflated 42%)
2022-09-27T16:55:56.7401716Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161921.xml (deflated 41%)
2022-09-27T16:55:56.7402487Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161927.xml (deflated 40%)
2022-09-27T16:55:56.7403255Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161936.xml (deflated 41%)
2022-09-27T16:55:56.7404026Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161944.xml (deflated 40%)
2022-09-27T16:55:56.7404799Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161957.xml (deflated 40%)
2022-09-27T16:55:56.7405549Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162013.xml (deflated 41%)
2022-09-27T16:55:56.7406316Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162023.xml (deflated 41%)
2022-09-27T16:55:56.7407084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162025.xml (deflated 42%)
2022-09-27T16:55:56.7407843Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162033.xml (deflated 64%)
2022-09-27T16:55:56.7408665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162154.xml (deflated 41%)
2022-09-27T16:55:56.7409453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162202.xml (deflated 41%)
2022-09-27T16:55:56.7410231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162209.xml (deflated 41%)
2022-09-27T16:55:56.7411001Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162219.xml (deflated 40%)
2022-09-27T16:55:56.7411752Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162227.xml (deflated 40%)
2022-09-27T16:55:56.7412520Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162236.xml (deflated 39%)
2022-09-27T16:55:56.7413285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162245.xml (deflated 40%)
2022-09-27T16:55:56.7414059Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162248.xml (deflated 40%)
2022-09-27T16:55:56.7414807Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162257.xml (deflated 40%)
2022-09-27T16:55:56.7415580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162306.xml (deflated 42%)
2022-09-27T16:55:56.7416344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162311.xml (deflated 41%)
2022-09-27T16:55:56.7417115Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162318.xml (deflated 41%)
2022-09-27T16:55:56.7417864Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162327.xml (deflated 41%)
2022-09-27T16:55:56.7418636Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162336.xml (deflated 41%)
2022-09-27T16:55:56.7419400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162339.xml (deflated 45%)
2022-09-27T16:55:56.7420248Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162346.xml (deflated 47%)
2022-09-27T16:55:56.7420997Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162353.xml (deflated 48%)
2022-09-27T16:55:56.7421766Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162400.xml (deflated 46%)
2022-09-27T16:55:56.7422527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162407.xml (deflated 40%)
2022-09-27T16:55:56.7423291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162415.xml (deflated 41%)
2022-09-27T16:55:56.7424047Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162422.xml (deflated 41%)
2022-09-27T16:55:56.7424812Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162429.xml (deflated 41%)
2022-09-27T16:55:56.7425580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162436.xml (deflated 41%)
2022-09-27T16:55:56.7426353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162443.xml (deflated 40%)
2022-09-27T16:55:56.7427103Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162450.xml (deflated 40%)
2022-09-27T16:55:56.7427918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162457.xml (deflated 41%)
2022-09-27T16:55:56.7428708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162500.xml (deflated 41%)
2022-09-27T16:55:56.7429476Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162503.xml (deflated 40%)
2022-09-27T16:55:56.7430222Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162510.xml (deflated 42%)
2022-09-27T16:55:56.7430986Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162516.xml (deflated 42%)
2022-09-27T16:55:56.7431748Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162519.xml (deflated 42%)
2022-09-27T16:55:56.7432515Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162522.xml (deflated 40%)
2022-09-27T16:55:56.7433267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162529.xml (deflated 39%)
2022-09-27T16:55:56.7434037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162536.xml (deflated 42%)
2022-09-27T16:55:56.7434800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162538.xml (deflated 40%)
2022-09-27T16:55:56.7435561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162546.xml (deflated 40%)
2022-09-27T16:55:56.7436313Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162553.xml (deflated 40%)
2022-09-27T16:55:56.7437078Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162600.xml (deflated 40%)
2022-09-27T16:55:56.7437846Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162607.xml (deflated 41%)
2022-09-27T16:55:56.7438611Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162614.xml (deflated 41%)
2022-09-27T16:55:56.7439431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162621.xml (deflated 42%)
2022-09-27T16:55:56.7440444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162628.xml (deflated 42%)
2022-09-27T16:55:56.7441296Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162635.xml (deflated 40%)
2022-09-27T16:55:56.7442065Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162642.xml (deflated 40%)
2022-09-27T16:55:56.7442815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162649.xml (deflated 41%)
2022-09-27T16:55:56.7443588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162656.xml (deflated 41%)
2022-09-27T16:55:56.7444359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162703.xml (deflated 40%)
2022-09-27T16:55:56.7445121Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162710.xml (deflated 40%)
2022-09-27T16:55:56.7445882Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162717.xml (deflated 40%)
2022-09-27T16:55:56.7446633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162725.xml (deflated 41%)
2022-09-27T16:55:56.7447487Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162732.xml (deflated 40%)
2022-09-27T16:55:56.7448270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162739.xml (deflated 40%)
2022-09-27T16:55:56.7449044Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162746.xml (deflated 42%)
2022-09-27T16:55:56.7449793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162754.xml (deflated 42%)
2022-09-27T16:55:56.7450562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162801.xml (deflated 41%)
2022-09-27T16:55:56.7451320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162808.xml (deflated 42%)
2022-09-27T16:55:56.7452084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162815.xml (deflated 40%)
2022-09-27T16:55:56.7452845Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162823.xml (deflated 40%)
2022-09-27T16:55:56.7453612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162830.xml (deflated 40%)
2022-09-27T16:55:56.7454384Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162838.xml (deflated 40%)
2022-09-27T16:55:56.7455150Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162847.xml (deflated 40%)
2022-09-27T16:55:56.7455896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162854.xml (deflated 40%)
2022-09-27T16:55:56.7456665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162903.xml (deflated 40%)
2022-09-27T16:55:56.7457434Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162910.xml (deflated 40%)
2022-09-27T16:55:56.7458201Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162917.xml (deflated 40%)
2022-09-27T16:55:56.7459042Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162924.xml (deflated 40%)
2022-09-27T16:55:56.7459856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162935.xml (deflated 40%)
2022-09-27T16:55:56.7460629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162946.xml (deflated 40%)
2022-09-27T16:55:56.7461401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162957.xml (deflated 42%)
2022-09-27T16:55:56.7462154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162959.xml (deflated 42%)
2022-09-27T16:55:56.7462925Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163002.xml (deflated 42%)
2022-09-27T16:55:56.7463690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163004.xml (deflated 41%)
2022-09-27T16:55:56.7464453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163007.xml (deflated 42%)
2022-09-27T16:55:56.7465201Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163009.xml (deflated 41%)
2022-09-27T16:55:56.7466020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163012.xml (deflated 42%)
2022-09-27T16:55:56.7466794Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163014.xml (deflated 42%)
2022-09-27T16:55:56.7467600Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163017.xml (deflated 42%)
2022-09-27T16:55:56.7468359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163019.xml (deflated 42%)
2022-09-27T16:55:56.7469118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163022.xml (deflated 42%)
2022-09-27T16:55:56.7469884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163024.xml (deflated 42%)
2022-09-27T16:55:56.7470648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163027.xml (deflated 42%)
2022-09-27T16:55:56.7471395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163029.xml (deflated 42%)
2022-09-27T16:55:56.7472167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163032.xml (deflated 42%)
2022-09-27T16:55:56.7472939Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163034.xml (deflated 42%)
2022-09-27T16:55:56.7473702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163037.xml (deflated 42%)
2022-09-27T16:55:56.7474446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163040.xml (deflated 42%)
2022-09-27T16:55:56.7475215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163042.xml (deflated 42%)
2022-09-27T16:55:56.7475977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163045.xml (deflated 42%)
2022-09-27T16:55:56.7476747Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163047.xml (deflated 42%)
2022-09-27T16:55:56.7477568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163050.xml (deflated 42%)
2022-09-27T16:55:56.7478336Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163052.xml (deflated 42%)
2022-09-27T16:55:56.7479093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163055.xml (deflated 42%)
2022-09-27T16:55:56.7479859Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163057.xml (deflated 41%)
2022-09-27T16:55:56.7481041Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163107.xml (deflated 41%)
2022-09-27T16:55:56.7481813Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163114.xml (deflated 42%)
2022-09-27T16:55:56.7482587Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163117.xml (deflated 40%)
2022-09-27T16:55:56.7483356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163125.xml (deflated 41%)
2022-09-27T16:55:56.7484109Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163135.xml (deflated 40%)
2022-09-27T16:55:56.7484862Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163143.xml (deflated 42%)
2022-09-27T16:55:56.7485716Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163150.xml (deflated 42%)
2022-09-27T16:55:56.7486500Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163157.xml (deflated 42%)
2022-09-27T16:55:56.7487305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163205.xml (deflated 40%)
2022-09-27T16:55:56.7488062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163213.xml (deflated 40%)
2022-09-27T16:55:56.7488831Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163221.xml (deflated 42%)
2022-09-27T16:55:56.7489599Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163228.xml (deflated 40%)
2022-09-27T16:55:56.7490366Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163235.xml (deflated 40%)
2022-09-27T16:55:56.7491111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163243.xml (deflated 41%)
2022-09-27T16:55:56.7491884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163245.xml (deflated 41%)
2022-09-27T16:55:56.7492659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163248.xml (deflated 40%)
2022-09-27T16:55:56.7493528Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163250.xml (deflated 41%)
2022-09-27T16:55:56.7494294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163253.xml (deflated 41%)
2022-09-27T16:55:56.7495061Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163255.xml (deflated 40%)
2022-09-27T16:55:56.7495812Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163258.xml (deflated 41%)
2022-09-27T16:55:56.7496579Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163300.xml (deflated 41%)
2022-09-27T16:55:56.7497451Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163302.xml (deflated 40%)
2022-09-27T16:55:56.7498215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163420.xml (deflated 42%)
2022-09-27T16:55:56.7498961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163424.xml (deflated 40%)
2022-09-27T16:55:56.7499732Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163431.xml (deflated 42%)
2022-09-27T16:55:56.7500499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163438.xml (deflated 40%)
2022-09-27T16:55:56.7501260Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163447.xml (deflated 42%)
2022-09-27T16:55:56.7502008Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163451.xml (deflated 42%)
2022-09-27T16:55:56.7502780Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163455.xml (deflated 42%)
2022-09-27T16:55:56.7503542Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163500.xml (deflated 40%)
2022-09-27T16:55:56.7504303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163509.xml (deflated 40%)
2022-09-27T16:55:56.7505117Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163518.xml (deflated 40%)
2022-09-27T16:55:56.7505895Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163527.xml (deflated 40%)
2022-09-27T16:55:56.7506661Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163536.xml (deflated 40%)
2022-09-27T16:55:56.7507424Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163545.xml (deflated 40%)
2022-09-27T16:55:56.7508169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163554.xml (deflated 40%)
2022-09-27T16:55:56.7508919Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163603.xml (deflated 40%)
2022-09-27T16:55:56.7509671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163612.xml (deflated 41%)
2022-09-27T16:55:56.7510418Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163622.xml (deflated 41%)
2022-09-27T16:55:56.7511158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163629.xml (deflated 41%)
2022-09-27T16:55:56.7511928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163639.xml (deflated 40%)
2022-09-27T16:55:56.7512692Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163648.xml (deflated 40%)
2022-09-27T16:55:56.7513449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163657.xml (deflated 40%)
2022-09-27T16:55:56.7514201Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163706.xml (deflated 40%)
2022-09-27T16:55:56.7514964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163716.xml (deflated 40%)
2022-09-27T16:55:56.7515718Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163726.xml (deflated 39%)
2022-09-27T16:55:56.7516534Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163736.xml (deflated 40%)
2022-09-27T16:55:56.7517287Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163746.xml (deflated 40%)
2022-09-27T16:55:56.7518021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163755.xml (deflated 41%)
2022-09-27T16:55:56.7518782Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163805.xml (deflated 41%)
2022-09-27T16:55:56.7519547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163824.xml (deflated 40%)
2022-09-27T16:55:56.7520645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163833.xml (deflated 40%)
2022-09-27T16:55:56.7521430Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163843.xml (deflated 40%)
2022-09-27T16:55:56.7522195Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163852.xml (deflated 40%)
2022-09-27T16:55:56.7522963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163902.xml (deflated 40%)
2022-09-27T16:55:56.7523811Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163911.xml (deflated 40%)
2022-09-27T16:55:56.7524577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163920.xml (deflated 41%)
2022-09-27T16:55:56.7525347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163930.xml (deflated 41%)
2022-09-27T16:55:56.7526116Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163939.xml (deflated 40%)
2022-09-27T16:55:56.7526883Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163949.xml (deflated 40%)
2022-09-27T16:55:56.7527632Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163958.xml (deflated 40%)
2022-09-27T16:55:56.7528402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164008.xml (deflated 40%)
2022-09-27T16:55:56.7529163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164017.xml (deflated 40%)
2022-09-27T16:55:56.7529927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164025.xml (deflated 40%)
2022-09-27T16:55:56.7530680Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164034.xml (deflated 40%)
2022-09-27T16:55:56.7531439Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164043.xml (deflated 40%)
2022-09-27T16:55:56.7532202Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164101.xml (deflated 41%)
2022-09-27T16:55:56.7532968Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164111.xml (deflated 41%)
2022-09-27T16:55:56.7533717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164120.xml (deflated 41%)
2022-09-27T16:55:56.7534480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164130.xml (deflated 40%)
2022-09-27T16:55:56.7535338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164139.xml (deflated 40%)
2022-09-27T16:55:56.7536107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164148.xml (deflated 42%)
2022-09-27T16:55:56.7536852Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164158.xml (deflated 41%)
2022-09-27T16:55:56.7537625Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164208.xml (deflated 40%)
2022-09-27T16:55:56.7538394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164217.xml (deflated 40%)
2022-09-27T16:55:56.7539162Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164226.xml (deflated 40%)
2022-09-27T16:55:56.7539912Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164235.xml (deflated 40%)
2022-09-27T16:55:56.7540683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164244.xml (deflated 41%)
2022-09-27T16:55:56.7541445Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164254.xml (deflated 40%)
2022-09-27T16:55:56.7542207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164303.xml (deflated 40%)
2022-09-27T16:55:56.7543015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164312.xml (deflated 40%)
2022-09-27T16:55:56.7543783Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164321.xml (deflated 42%)
2022-09-27T16:55:56.7544546Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164326.xml (deflated 42%)
2022-09-27T16:55:56.7545321Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164330.xml (deflated 40%)
2022-09-27T16:55:56.7546068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164350.xml (deflated 40%)
2022-09-27T16:55:56.7546835Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164358.xml (deflated 40%)
2022-09-27T16:55:56.7547607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164405.xml (deflated 40%)
2022-09-27T16:55:56.7548365Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164415.xml (deflated 41%)
2022-09-27T16:55:56.7549113Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164424.xml (deflated 40%)
2022-09-27T16:55:56.7549878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164431.xml (deflated 40%)
2022-09-27T16:55:56.7550642Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164439.xml (deflated 40%)
2022-09-27T16:55:56.7551400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164446.xml (deflated 41%)
2022-09-27T16:55:56.7552170Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164448.xml (deflated 40%)
2022-09-27T16:55:56.7552919Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164455.xml (deflated 41%)
2022-09-27T16:55:56.7553685Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164503.xml (deflated 40%)
2022-09-27T16:55:56.7554524Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164510.xml (deflated 42%)
2022-09-27T16:55:56.7555288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164514.xml (deflated 40%)
2022-09-27T16:55:56.7556034Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164521.xml (deflated 40%)
2022-09-27T16:55:56.7556801Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164528.xml (deflated 40%)
2022-09-27T16:55:56.7557570Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164535.xml (deflated 40%)
2022-09-27T16:55:56.7558335Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164542.xml (deflated 41%)
2022-09-27T16:55:56.7559081Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164549.xml (deflated 40%)
2022-09-27T16:55:56.7559885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164559.xml (deflated 40%)
2022-09-27T16:55:56.7561016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164606.xml (deflated 40%)
2022-09-27T16:55:56.7561787Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164613.xml (deflated 40%)
2022-09-27T16:55:56.7562616Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164621.xml (deflated 40%)
2022-09-27T16:55:56.7563406Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164628.xml (deflated 40%)
2022-09-27T16:55:56.7564169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164654.xml (deflated 40%)
2022-09-27T16:55:56.7564933Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164720.xml (deflated 42%)
2022-09-27T16:55:56.7565682Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164727.xml (deflated 40%)
2022-09-27T16:55:56.7566442Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164736.xml (deflated 40%)
2022-09-27T16:55:56.7567216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164743.xml (deflated 40%)
2022-09-27T16:55:56.7567980Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164750.xml (deflated 41%)
2022-09-27T16:55:56.7568727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164757.xml (deflated 42%)
2022-09-27T16:55:56.7569499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164759.xml (deflated 41%)
2022-09-27T16:55:56.7570270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164802.xml (deflated 42%)
2022-09-27T16:55:56.7571032Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164804.xml (deflated 41%)
2022-09-27T16:55:56.7571786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164807.xml (deflated 42%)
2022-09-27T16:55:56.7572558Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164809.xml (deflated 42%)
2022-09-27T16:55:56.7573327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164812.xml (deflated 41%)
2022-09-27T16:55:56.7574181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164814.xml (deflated 41%)
2022-09-27T16:55:56.7574931Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164816.xml (deflated 40%)
2022-09-27T16:55:56.7575696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164823.xml (deflated 40%)
2022-09-27T16:55:56.7576467Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164830.xml (deflated 41%)
2022-09-27T16:55:56.7577236Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164833.xml (deflated 42%)
2022-09-27T16:55:56.7577979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164835.xml (deflated 42%)
2022-09-27T16:55:56.7578743Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164839.xml (deflated 40%)
2022-09-27T16:55:56.7579511Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164848.xml (deflated 40%)
2022-09-27T16:55:56.7580272Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164857.xml (deflated 40%)
2022-09-27T16:55:56.7581066Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164907.xml (deflated 39%)
2022-09-27T16:55:56.7581846Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164916.xml (deflated 39%)
2022-09-27T16:55:56.7582612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164925.xml (deflated 40%)
2022-09-27T16:55:56.7583376Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164931.xml (deflated 40%)
2022-09-27T16:55:56.7584121Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164938.xml (deflated 39%)
2022-09-27T16:55:56.7584888Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164947.xml (deflated 40%)
2022-09-27T16:55:56.7585648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164954.xml (deflated 40%)
2022-09-27T16:55:56.7586415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165001.xml (deflated 40%)
2022-09-27T16:55:56.7587188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165008.xml (deflated 40%)
2022-09-27T16:55:56.7587939Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165015.xml (deflated 42%)
2022-09-27T16:55:56.7588703Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165022.xml (deflated 42%)
2022-09-27T16:55:56.7589469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165028.xml (deflated 42%)
2022-09-27T16:55:56.7590231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165036.xml (deflated 42%)
2022-09-27T16:55:56.7590981Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165042.xml (deflated 40%)
2022-09-27T16:55:56.7591747Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165049.xml (deflated 41%)
2022-09-27T16:55:56.7592505Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165055.xml (deflated 42%)
2022-09-27T16:55:56.7593348Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165058.xml (deflated 41%)
2022-09-27T16:55:56.7594095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165104.xml (deflated 43%)
2022-09-27T16:55:56.7594859Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165107.xml (deflated 40%)
2022-09-27T16:55:56.7595623Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165113.xml (deflated 42%)
2022-09-27T16:55:56.7596392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165116.xml (deflated 42%)
2022-09-27T16:55:56.7597136Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165118.xml (deflated 40%)
2022-09-27T16:55:56.7597902Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165124.xml (deflated 40%)
2022-09-27T16:55:56.7598670Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165131.xml (deflated 41%)
2022-09-27T16:55:56.7599436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165138.xml (deflated 40%)
2022-09-27T16:55:56.7600521Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165145.xml (deflated 41%)
2022-09-27T16:55:56.7601380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165147.xml (deflated 41%)
2022-09-27T16:55:56.7602154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165149.xml (deflated 40%)
2022-09-27T16:55:56.7602926Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165156.xml (deflated 42%)
2022-09-27T16:55:56.7603677Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165203.xml (deflated 40%)
2022-09-27T16:55:56.7604441Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165209.xml (deflated 40%)
2022-09-27T16:55:56.7605207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165216.xml (deflated 41%)
2022-09-27T16:55:56.7605977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165222.xml (deflated 40%)
2022-09-27T16:55:56.7606728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165229.xml (deflated 40%)
2022-09-27T16:55:56.7607499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165236.xml (deflated 40%)
2022-09-27T16:55:56.7608263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165242.xml (deflated 41%)
2022-09-27T16:55:56.7609027Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165245.xml (deflated 40%)
2022-09-27T16:55:56.7609784Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165247.xml (deflated 41%)
2022-09-27T16:55:56.7610552Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165249.xml (deflated 40%)
2022-09-27T16:55:56.7611316Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165256.xml (deflated 41%)
2022-09-27T16:55:56.7612181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165303.xml (deflated 40%)
2022-09-27T16:55:56.7612932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165309.xml (deflated 40%)
2022-09-27T16:55:56.7613700Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165316.xml (deflated 40%)
2022-09-27T16:55:56.7614465Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165323.xml (deflated 40%)
2022-09-27T16:55:56.7615234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165331.xml (deflated 41%)
2022-09-27T16:55:56.7615983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165339.xml (deflated 40%)
2022-09-27T16:55:56.7616758Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165346.xml (deflated 40%)
2022-09-27T16:55:56.7617521Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165354.xml (deflated 41%)
2022-09-27T16:55:56.7618286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165402.xml (deflated 40%)
2022-09-27T16:55:56.7619033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165416.xml (deflated 40%)
2022-09-27T16:55:56.7619866Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154655.xml (deflated 41%)
2022-09-27T16:55:56.7620660Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154706.xml (deflated 41%)
2022-09-27T16:55:56.7621440Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154708.xml (deflated 41%)
2022-09-27T16:55:56.7622208Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154716.xml (deflated 41%)
2022-09-27T16:55:56.7622981Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154834.xml (deflated 41%)
2022-09-27T16:55:56.7623748Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154843.xml (deflated 40%)
2022-09-27T16:55:56.7624534Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154853.xml (deflated 40%)
2022-09-27T16:55:56.7625288Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154903.xml (deflated 40%)
2022-09-27T16:55:56.7626056Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154912.xml (deflated 40%)
2022-09-27T16:55:56.7626836Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154921.xml (deflated 39%)
2022-09-27T16:55:56.7627608Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154931.xml (deflated 39%)
2022-09-27T16:55:56.7628382Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154933.xml (deflated 40%)
2022-09-27T16:55:56.7629142Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154943.xml (deflated 40%)
2022-09-27T16:55:56.7629908Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154952.xml (deflated 42%)
2022-09-27T16:55:56.7630670Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154956.xml (deflated 41%)
2022-09-27T16:55:56.7631530Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155004.xml (deflated 41%)
2022-09-27T16:55:56.7632291Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155014.xml (deflated 41%)
2022-09-27T16:55:56.7633069Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155023.xml (deflated 42%)
2022-09-27T16:55:56.7633832Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155025.xml (deflated 45%)
2022-09-27T16:55:56.7634602Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155028.xml (deflated 46%)
2022-09-27T16:55:56.7635355Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155030.xml (deflated 48%)
2022-09-27T16:55:56.7636124Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155033.xml (deflated 45%)
2022-09-27T16:55:56.7636886Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155035.xml (deflated 40%)
2022-09-27T16:55:56.7637659Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155045.xml (deflated 43%)
2022-09-27T16:55:56.7638408Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155048.xml (deflated 43%)
2022-09-27T16:55:56.7639225Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155051.xml (deflated 43%)
2022-09-27T16:55:56.7640009Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155053.xml (deflated 43%)
2022-09-27T16:55:56.7641127Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155056.xml (deflated 43%)
2022-09-27T16:55:56.7641882Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155058.xml (deflated 40%)
2022-09-27T16:55:56.7642654Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155109.xml (deflated 41%)
2022-09-27T16:55:56.7643421Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155111.xml (deflated 41%)
2022-09-27T16:55:56.7644185Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155114.xml (deflated 41%)
2022-09-27T16:55:56.7644937Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155124.xml (deflated 42%)
2022-09-27T16:55:56.7645705Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155131.xml (deflated 42%)
2022-09-27T16:55:56.7646472Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155134.xml (deflated 42%)
2022-09-27T16:55:56.7647238Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155136.xml (deflated 40%)
2022-09-27T16:55:56.7647990Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155139.xml (deflated 40%)
2022-09-27T16:55:56.7648762Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155142.xml (deflated 43%)
2022-09-27T16:55:56.7649526Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155144.xml (deflated 40%)
2022-09-27T16:55:56.7650296Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155147.xml (deflated 41%)
2022-09-27T16:55:56.7651150Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155149.xml (deflated 41%)
2022-09-27T16:55:56.7651913Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155152.xml (deflated 41%)
2022-09-27T16:55:56.7652679Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155154.xml (deflated 40%)
2022-09-27T16:55:56.7653456Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155157.xml (deflated 41%)
2022-09-27T16:55:56.7654209Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155159.xml (deflated 41%)
2022-09-27T16:55:56.7654973Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155202.xml (deflated 41%)
2022-09-27T16:55:56.7655745Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155205.xml (deflated 41%)
2022-09-27T16:55:56.7656510Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155207.xml (deflated 40%)
2022-09-27T16:55:56.7657260Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155214.xml (deflated 41%)
2022-09-27T16:55:56.7658091Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155217.xml (deflated 41%)
2022-09-27T16:55:56.7658874Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155219.xml (deflated 41%)
2022-09-27T16:55:56.7659642Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155222.xml (deflated 40%)
2022-09-27T16:55:56.7660440Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155229.xml (deflated 40%)
2022-09-27T16:55:56.7661211Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155239.xml (deflated 40%)
2022-09-27T16:55:56.7661976Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155249.xml (deflated 40%)
2022-09-27T16:55:56.7662743Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155300.xml (deflated 40%)
2022-09-27T16:55:56.7663506Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155310.xml (deflated 42%)
2022-09-27T16:55:56.7664278Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155317.xml (deflated 42%)
2022-09-27T16:55:56.7665048Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155324.xml (deflated 41%)
2022-09-27T16:55:56.7665814Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155331.xml (deflated 42%)
2022-09-27T16:55:56.7666557Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155338.xml (deflated 40%)
2022-09-27T16:55:56.7667323Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155348.xml (deflated 40%)
2022-09-27T16:55:56.7668098Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155359.xml (deflated 40%)
2022-09-27T16:55:56.7668867Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155408.xml (deflated 40%)
2022-09-27T16:55:56.7669633Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155418.xml (deflated 40%)
2022-09-27T16:55:56.7670454Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155428.xml (deflated 41%)
2022-09-27T16:55:56.7671222Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155431.xml (deflated 40%)
2022-09-27T16:55:56.7671990Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155442.xml (deflated 40%)
2022-09-27T16:55:56.7672755Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155452.xml (deflated 40%)
2022-09-27T16:55:56.7673512Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155502.xml (deflated 41%)
2022-09-27T16:55:56.7674282Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155504.xml (deflated 41%)
2022-09-27T16:55:56.7675051Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155507.xml (deflated 40%)
2022-09-27T16:55:56.7675817Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155509.xml (deflated 42%)
2022-09-27T16:55:56.7676571Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155512.xml (deflated 42%)
2022-09-27T16:55:56.7677380Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155514.xml (deflated 42%)
2022-09-27T16:55:56.7678163Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155517.xml (deflated 41%)
2022-09-27T16:55:56.7678931Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155519.xml (deflated 42%)
2022-09-27T16:55:56.7679679Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155522.xml (deflated 41%)
2022-09-27T16:55:56.7680820Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155524.xml (deflated 42%)
2022-09-27T16:55:56.7681607Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155527.xml (deflated 42%)
2022-09-27T16:55:56.7682380Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155530.xml (deflated 42%)
2022-09-27T16:55:56.7683137Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155532.xml (deflated 42%)
2022-09-27T16:55:56.7683912Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155535.xml (deflated 42%)
2022-09-27T16:55:56.7684683Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155537.xml (deflated 42%)
2022-09-27T16:55:56.7685446Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155539.xml (deflated 42%)
2022-09-27T16:55:56.7686199Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155542.xml (deflated 43%)
2022-09-27T16:55:56.7686966Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155544.xml (deflated 42%)
2022-09-27T16:55:56.7687734Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155547.xml (deflated 42%)
2022-09-27T16:55:56.7688510Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155549.xml (deflated 42%)
2022-09-27T16:55:56.7689365Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155552.xml (deflated 42%)
2022-09-27T16:55:56.7690134Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155554.xml (deflated 42%)
2022-09-27T16:55:56.7691070Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155557.xml (deflated 42%)
2022-09-27T16:55:56.7691840Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155559.xml (deflated 42%)
2022-09-27T16:55:56.7692593Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155602.xml (deflated 42%)
2022-09-27T16:55:56.7693361Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155605.xml (deflated 42%)
2022-09-27T16:55:56.7694123Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155607.xml (deflated 42%)
2022-09-27T16:55:56.7694892Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155610.xml (deflated 41%)
2022-09-27T16:55:56.7695644Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155620.xml (deflated 41%)
2022-09-27T16:55:56.7696410Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155623.xml (deflated 41%)
2022-09-27T16:55:56.7697244Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155625.xml (deflated 42%)
2022-09-27T16:55:56.7698018Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155628.xml (deflated 41%)
2022-09-27T16:55:56.7698775Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155630.xml (deflated 42%)
2022-09-27T16:55:56.7699546Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155633.xml (deflated 42%)
2022-09-27T16:55:56.7700316Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155640.xml (deflated 42%)
2022-09-27T16:55:56.7701073Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155643.xml (deflated 42%)
2022-09-27T16:55:56.7701825Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155650.xml (deflated 42%)
2022-09-27T16:55:56.7702592Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155652.xml (deflated 42%)
2022-09-27T16:55:56.7703360Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155655.xml (deflated 42%)
2022-09-27T16:55:56.7704127Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155657.xml (deflated 40%)
2022-09-27T16:55:56.7704876Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155700.xml (deflated 40%)
2022-09-27T16:55:56.7705622Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155702.xml (deflated 41%)
2022-09-27T16:55:56.7706371Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155705.xml (deflated 41%)
2022-09-27T16:55:56.7707112Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155707.xml (deflated 40%)
2022-09-27T16:55:56.7707859Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155710.xml (deflated 41%)
2022-09-27T16:55:56.7708674Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155712.xml (deflated 41%)
2022-09-27T16:55:56.7709427Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155715.xml (deflated 40%)
2022-09-27T16:55:56.7710172Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155717.xml (deflated 41%)
2022-09-27T16:55:56.7710918Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155720.xml (deflated 41%)
2022-09-27T16:55:56.7711665Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155722.xml (deflated 40%)
2022-09-27T16:55:56.7712417Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155732.xml (deflated 41%)
2022-09-27T16:55:56.7713191Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155735.xml (deflated 40%)
2022-09-27T16:55:56.7713952Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155745.xml (deflated 42%)
2022-09-27T16:55:56.7714703Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155752.xml (deflated 40%)
2022-09-27T16:55:56.7715469Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155801.xml (deflated 40%)
2022-09-27T16:55:56.7716286Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155804.xml (deflated 40%)
2022-09-27T16:55:56.7717066Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155806.xml (deflated 40%)
2022-09-27T16:55:56.7717814Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155809.xml (deflated 40%)
2022-09-27T16:55:56.7718589Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155818.xml (deflated 40%)
2022-09-27T16:55:56.7719352Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155827.xml (deflated 40%)
2022-09-27T16:55:56.7720118Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155837.xml (deflated 40%)
2022-09-27T16:55:56.7721051Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155847.xml (deflated 40%)
2022-09-27T16:55:56.7721821Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155849.xml (deflated 40%)
2022-09-27T16:55:56.7722588Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155858.xml (deflated 40%)
2022-09-27T16:55:56.7723364Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155908.xml (deflated 40%)
2022-09-27T16:55:56.7724115Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155910.xml (deflated 41%)
2022-09-27T16:55:56.7724880Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155913.xml (deflated 40%)
2022-09-27T16:55:56.7725642Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155922.xml (deflated 41%)
2022-09-27T16:55:56.7726407Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155924.xml (deflated 40%)
2022-09-27T16:55:56.7727159Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155927.xml (deflated 40%)
2022-09-27T16:55:56.7728026Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155937.xml (deflated 40%)
2022-09-27T16:55:56.7728795Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155946.xml (deflated 40%)
2022-09-27T16:55:56.7729564Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155956.xml (deflated 40%)
2022-09-27T16:55:56.7730317Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160006.xml (deflated 40%)
2022-09-27T16:55:56.7731089Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160017.xml (deflated 40%)
2022-09-27T16:55:56.7731856Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160019.xml (deflated 42%)
2022-09-27T16:55:56.7732622Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160022.xml (deflated 42%)
2022-09-27T16:55:56.7733375Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160024.xml (deflated 42%)
2022-09-27T16:55:56.7734140Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160027.xml (deflated 41%)
2022-09-27T16:55:56.7734905Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160029.xml (deflated 41%)
2022-09-27T16:55:56.7735792Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160032.xml (deflated 41%)
2022-09-27T16:55:56.7736556Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160034.xml (deflated 41%)
2022-09-27T16:55:56.7737331Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160037.xml (deflated 41%)
2022-09-27T16:55:56.7738105Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160040.xml (deflated 41%)
2022-09-27T16:55:56.7738873Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160042.xml (deflated 41%)
2022-09-27T16:55:56.7739622Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160045.xml (deflated 41%)
2022-09-27T16:55:56.7740386Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160047.xml (deflated 42%)
2022-09-27T16:55:56.7741152Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160050.xml (deflated 41%)
2022-09-27T16:55:56.7741916Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160052.xml (deflated 40%)
2022-09-27T16:55:56.7742668Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160102.xml (deflated 40%)
2022-09-27T16:55:56.7743435Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160112.xml (deflated 40%)
2022-09-27T16:55:56.7744200Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160122.xml (deflated 40%)
2022-09-27T16:55:56.7744968Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160131.xml (deflated 40%)
2022-09-27T16:55:56.7745737Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160134.xml (deflated 40%)
2022-09-27T16:55:56.7746483Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160136.xml (deflated 41%)
2022-09-27T16:55:56.7747314Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160139.xml (deflated 41%)
2022-09-27T16:55:56.7748070Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160141.xml (deflated 41%)
2022-09-27T16:55:56.7748843Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160144.xml (deflated 41%)
2022-09-27T16:55:56.7749594Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160154.xml (deflated 40%)
2022-09-27T16:55:56.7750358Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160203.xml (deflated 42%)
2022-09-27T16:55:56.7751117Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160206.xml (deflated 42%)
2022-09-27T16:55:56.7751868Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160209.xml (deflated 40%)
2022-09-27T16:55:56.7752614Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160218.xml (deflated 40%)
2022-09-27T16:55:56.7753385Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160227.xml (deflated 41%)
2022-09-27T16:55:56.7754202Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160230.xml (deflated 41%)
2022-09-27T16:55:56.7754979Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160239.xml (deflated 41%)
2022-09-27T16:55:56.7755733Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160250.xml (deflated 41%)
2022-09-27T16:55:56.7756497Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160259.xml (deflated 40%)
2022-09-27T16:55:56.7757267Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160309.xml (deflated 40%)
2022-09-27T16:55:56.7758034Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160318.xml (deflated 42%)
2022-09-27T16:55:56.7758781Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160323.xml (deflated 42%)
2022-09-27T16:55:56.7759552Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160327.xml (deflated 40%)
2022-09-27T16:55:56.7760369Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160330.xml (deflated 40%)
2022-09-27T16:55:56.7761292Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160348.xml (deflated 40%)
2022-09-27T16:55:56.7762052Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160355.xml (deflated 40%)
2022-09-27T16:55:56.7762826Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160405.xml (deflated 41%)
2022-09-27T16:55:56.7763596Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160408.xml (deflated 40%)
2022-09-27T16:55:56.7764360Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160415.xml (deflated 41%)
2022-09-27T16:55:56.7765111Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160417.xml (deflated 40%)
2022-09-27T16:55:56.7765882Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160420.xml (deflated 41%)
2022-09-27T16:55:56.7766757Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160422.xml (deflated 40%)
2022-09-27T16:55:56.7767530Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160425.xml (deflated 41%)
2022-09-27T16:55:56.7768329Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160427.xml (deflated 41%)
2022-09-27T16:55:56.7769103Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160430.xml (deflated 40%)
2022-09-27T16:55:56.7769874Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160433.xml (deflated 41%)
2022-09-27T16:55:56.7770642Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160440.xml (deflated 42%)
2022-09-27T16:55:56.7771396Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160443.xml (deflated 40%)
2022-09-27T16:55:56.7772168Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160450.xml (deflated 40%)
2022-09-27T16:55:56.7772937Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160457.xml (deflated 40%)
2022-09-27T16:55:56.7773764Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160504.xml (deflated 41%)
2022-09-27T16:55:56.7774530Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160507.xml (deflated 40%)
2022-09-27T16:55:56.7775305Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160516.xml (deflated 41%)
2022-09-27T16:55:56.7776076Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160525.xml (deflated 41%)
2022-09-27T16:55:56.7776842Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160535.xml (deflated 40%)
2022-09-27T16:55:56.7777591Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160546.xml (deflated 40%)
2022-09-27T16:55:56.7778359Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160548.xml (deflated 40%)
2022-09-27T16:55:56.7779128Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160551.xml (deflated 42%)
2022-09-27T16:55:56.7779898Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160553.xml (deflated 41%)
2022-09-27T16:55:56.7780652Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160556.xml (deflated 41%)
2022-09-27T16:55:56.7781425Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160558.xml (deflated 41%)
2022-09-27T16:55:56.7782188Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160601.xml (deflated 41%)
2022-09-27T16:55:56.7782955Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160603.xml (deflated 42%)
2022-09-27T16:55:56.7783703Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160606.xml (deflated 42%)
2022-09-27T16:55:56.7784469Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160608.xml (deflated 42%)
2022-09-27T16:55:56.7785237Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160611.xml (deflated 41%)
2022-09-27T16:55:56.7786064Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160613.xml (deflated 42%)
2022-09-27T16:55:56.7786833Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160616.xml (deflated 42%)
2022-09-27T16:55:56.7787588Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160618.xml (deflated 41%)
2022-09-27T16:55:56.7788361Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160621.xml (deflated 41%)
2022-09-27T16:55:56.7789131Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160623.xml (deflated 40%)
2022-09-27T16:55:56.7789903Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160630.xml (deflated 40%)
2022-09-27T16:55:56.7790662Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160637.xml (deflated 41%)
2022-09-27T16:55:56.7791433Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160640.xml (deflated 42%)
2022-09-27T16:55:56.7792202Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160642.xml (deflated 42%)
2022-09-27T16:55:56.7793016Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160647.xml (deflated 40%)
2022-09-27T16:55:56.7793777Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160656.xml (deflated 41%)
2022-09-27T16:55:56.7794550Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160707.xml (deflated 40%)
2022-09-27T16:55:56.7795321Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160717.xml (deflated 40%)
2022-09-27T16:55:56.7796089Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160727.xml (deflated 39%)
2022-09-27T16:55:56.7796838Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160737.xml (deflated 40%)
2022-09-27T16:55:56.7797614Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160744.xml (deflated 40%)
2022-09-27T16:55:56.7798381Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160751.xml (deflated 64%)
2022-09-27T16:55:56.7799146Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160810.xml (deflated 40%)
2022-09-27T16:55:56.7799902Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160820.xml (deflated 40%)
2022-09-27T16:55:56.7800838Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160830.xml (deflated 40%)
2022-09-27T16:55:56.7801610Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160840.xml (deflated 40%)
2022-09-27T16:55:56.7802371Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160850.xml (deflated 42%)
2022-09-27T16:55:56.7803126Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160858.xml (deflated 42%)
2022-09-27T16:55:56.7803894Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160905.xml (deflated 42%)
2022-09-27T16:55:56.7804753Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160911.xml (deflated 42%)
2022-09-27T16:55:56.7805521Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160918.xml (deflated 40%)
2022-09-27T16:55:56.7806272Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160928.xml (deflated 41%)
2022-09-27T16:55:56.7807042Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160939.xml (deflated 42%)
2022-09-27T16:55:56.7807811Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160941.xml (deflated 41%)
2022-09-27T16:55:56.7808575Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160951.xml (deflated 43%)
2022-09-27T16:55:56.7809324Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160954.xml (deflated 40%)
2022-09-27T16:55:56.7810092Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161004.xml (deflated 42%)
2022-09-27T16:55:56.7810849Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161007.xml (deflated 42%)
2022-09-27T16:55:56.7811613Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161009.xml (deflated 40%)
2022-09-27T16:55:56.7812425Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161019.xml (deflated 41%)
2022-09-27T16:55:56.7813195Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161022.xml (deflated 41%)
2022-09-27T16:55:56.7813959Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161024.xml (deflated 41%)
2022-09-27T16:55:56.7814732Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161027.xml (deflated 41%)
2022-09-27T16:55:56.7815475Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161029.xml (deflated 41%)
2022-09-27T16:55:56.7816242Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161032.xml (deflated 41%)
2022-09-27T16:55:56.7817005Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161035.xml (deflated 41%)
2022-09-27T16:55:56.7817768Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161037.xml (deflated 41%)
2022-09-27T16:55:56.7818513Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161040.xml (deflated 41%)
2022-09-27T16:55:56.7819286Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161049.xml (deflated 42%)
2022-09-27T16:55:56.7820049Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161052.xml (deflated 42%)
2022-09-27T16:55:56.7820811Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161054.xml (deflated 42%)
2022-09-27T16:55:56.7821562Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161057.xml (deflated 40%)
2022-09-27T16:55:56.7822332Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161107.xml (deflated 41%)
2022-09-27T16:55:56.7823094Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161110.xml (deflated 41%)
2022-09-27T16:55:56.7823932Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161112.xml (deflated 41%)
2022-09-27T16:55:56.7824704Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161120.xml (deflated 41%)
2022-09-27T16:55:56.7825453Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161135.xml (deflated 41%)
2022-09-27T16:55:56.7826215Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161144.xml (deflated 40%)
2022-09-27T16:55:56.7826987Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161154.xml (deflated 40%)
2022-09-27T16:55:56.7827751Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161205.xml (deflated 42%)
2022-09-27T16:55:56.7828498Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161207.xml (deflated 42%)
2022-09-27T16:55:56.7829261Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161210.xml (deflated 42%)
2022-09-27T16:55:56.7830020Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161219.xml (deflated 41%)
2022-09-27T16:55:56.7830785Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161228.xml (deflated 41%)
2022-09-27T16:55:56.7831585Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161230.xml (deflated 41%)
2022-09-27T16:55:56.7832368Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161240.xml (deflated 40%)
2022-09-27T16:55:56.7833128Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161242.xml (deflated 40%)
2022-09-27T16:55:56.7833900Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162013.xml (deflated 41%)
2022-09-27T16:55:56.7834652Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162023.xml (deflated 41%)
2022-09-27T16:55:56.7835416Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162025.xml (deflated 42%)
2022-09-27T16:55:56.7836184Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162033.xml (deflated 41%)
2022-09-27T16:55:56.7836958Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162151.xml (deflated 41%)
2022-09-27T16:55:56.7837705Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162200.xml (deflated 41%)
2022-09-27T16:55:56.7838478Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162209.xml (deflated 41%)
2022-09-27T16:55:56.7839243Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162220.xml (deflated 40%)
2022-09-27T16:55:56.7840009Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162229.xml (deflated 40%)
2022-09-27T16:55:56.7840990Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162238.xml (deflated 39%)
2022-09-27T16:55:56.7841767Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162248.xml (deflated 40%)
2022-09-27T16:55:56.7842532Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162250.xml (deflated 40%)
2022-09-27T16:55:56.7843388Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162300.xml (deflated 40%)
2022-09-27T16:55:56.7844142Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162309.xml (deflated 42%)
2022-09-27T16:55:56.7844911Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162314.xml (deflated 41%)
2022-09-27T16:55:56.7845688Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162320.xml (deflated 41%)
2022-09-27T16:55:56.7846465Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162330.xml (deflated 41%)
2022-09-27T16:55:56.7847214Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162339.xml (deflated 41%)
2022-09-27T16:55:56.7847985Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162342.xml (deflated 45%)
2022-09-27T16:55:56.7848751Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162344.xml (deflated 46%)
2022-09-27T16:55:56.7849515Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162347.xml (deflated 48%)
2022-09-27T16:55:56.7850262Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162349.xml (deflated 45%)
2022-09-27T16:55:56.7851102Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162352.xml (deflated 40%)
2022-09-27T16:55:56.7851892Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162402.xml (deflated 43%)
2022-09-27T16:55:56.7852661Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162405.xml (deflated 43%)
2022-09-27T16:55:56.7853409Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162407.xml (deflated 43%)
2022-09-27T16:55:56.7854178Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162410.xml (deflated 43%)
2022-09-27T16:55:56.7854938Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162412.xml (deflated 43%)
2022-09-27T16:55:56.7855706Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162415.xml (deflated 40%)
2022-09-27T16:55:56.7856463Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162425.xml (deflated 41%)
2022-09-27T16:55:56.7857226Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162428.xml (deflated 41%)
2022-09-27T16:55:56.7857990Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162430.xml (deflated 41%)
2022-09-27T16:55:56.7858757Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162441.xml (deflated 42%)
2022-09-27T16:55:56.7859504Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162448.xml (deflated 42%)
2022-09-27T16:55:56.7860316Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162450.xml (deflated 42%)
2022-09-27T16:55:56.7861093Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162453.xml (deflated 40%)
2022-09-27T16:55:56.7861857Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162455.xml (deflated 40%)
2022-09-27T16:55:56.7862675Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162458.xml (deflated 42%)
2022-09-27T16:55:56.7863445Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162500.xml (deflated 41%)
2022-09-27T16:55:56.7864212Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162503.xml (deflated 41%)
2022-09-27T16:55:56.7864984Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162505.xml (deflated 41%)
2022-09-27T16:55:56.7865753Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162508.xml (deflated 41%)
2022-09-27T16:55:56.7866510Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162510.xml (deflated 41%)
2022-09-27T16:55:56.7867283Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162513.xml (deflated 41%)
2022-09-27T16:55:56.7868049Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162516.xml (deflated 41%)
2022-09-27T16:55:56.7868817Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162518.xml (deflated 41%)
2022-09-27T16:55:56.7869623Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162521.xml (deflated 41%)
2022-09-27T16:55:56.7870399Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162523.xml (deflated 40%)
2022-09-27T16:55:56.7871166Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162530.xml (deflated 41%)
2022-09-27T16:55:56.7871941Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162532.xml (deflated 41%)
2022-09-27T16:55:56.7872680Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162535.xml (deflated 41%)
2022-09-27T16:55:56.7873445Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162537.xml (deflated 40%)
2022-09-27T16:55:56.7874209Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162545.xml (deflated 40%)
2022-09-27T16:55:56.7874981Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162555.xml (deflated 40%)
2022-09-27T16:55:56.7875732Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162605.xml (deflated 40%)
2022-09-27T16:55:56.7876502Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162615.xml (deflated 40%)
2022-09-27T16:55:56.7877262Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162625.xml (deflated 41%)
2022-09-27T16:55:56.7878029Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162633.xml (deflated 42%)
2022-09-27T16:55:56.7878777Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162640.xml (deflated 42%)
2022-09-27T16:55:56.7879530Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162647.xml (deflated 42%)
2022-09-27T16:55:56.7880522Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162654.xml (deflated 41%)
2022-09-27T16:55:56.7881304Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162704.xml (deflated 40%)
2022-09-27T16:55:56.7882151Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162714.xml (deflated 40%)
2022-09-27T16:55:56.7882925Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162724.xml (deflated 41%)
2022-09-27T16:55:56.7883690Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162734.xml (deflated 40%)
2022-09-27T16:55:56.7884454Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162744.xml (deflated 41%)
2022-09-27T16:55:56.7885198Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162747.xml (deflated 40%)
2022-09-27T16:55:56.7885963Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162757.xml (deflated 40%)
2022-09-27T16:55:56.7886734Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162807.xml (deflated 40%)
2022-09-27T16:55:56.7887492Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162817.xml (deflated 40%)
2022-09-27T16:55:56.7888238Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162820.xml (deflated 40%)
2022-09-27T16:55:56.7889061Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162822.xml (deflated 40%)
2022-09-27T16:55:56.7889838Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162825.xml (deflated 42%)
2022-09-27T16:55:56.7890596Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162827.xml (deflated 42%)
2022-09-27T16:55:56.7891348Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162830.xml (deflated 42%)
2022-09-27T16:55:56.7892119Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162832.xml (deflated 41%)
2022-09-27T16:55:56.7892885Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162835.xml (deflated 42%)
2022-09-27T16:55:56.7893650Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162837.xml (deflated 41%)
2022-09-27T16:55:56.7894400Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162840.xml (deflated 42%)
2022-09-27T16:55:56.7895158Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162842.xml (deflated 41%)
2022-09-27T16:55:56.7895928Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162845.xml (deflated 42%)
2022-09-27T16:55:56.7896693Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162848.xml (deflated 42%)
2022-09-27T16:55:56.7897442Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162850.xml (deflated 42%)
2022-09-27T16:55:56.7898207Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162853.xml (deflated 42%)
2022-09-27T16:55:56.7898976Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162855.xml (deflated 42%)
2022-09-27T16:55:56.7899743Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162858.xml (deflated 42%)
2022-09-27T16:55:56.7900574Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162900.xml (deflated 42%)
2022-09-27T16:55:56.7901346Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162903.xml (deflated 42%)
2022-09-27T16:55:56.7902120Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162905.xml (deflated 42%)
2022-09-27T16:55:56.7902880Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162908.xml (deflated 42%)
2022-09-27T16:55:56.7903632Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162910.xml (deflated 42%)
2022-09-27T16:55:56.7904394Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162913.xml (deflated 42%)
2022-09-27T16:55:56.7905157Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162915.xml (deflated 42%)
2022-09-27T16:55:56.7905921Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162918.xml (deflated 42%)
2022-09-27T16:55:56.7906686Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162920.xml (deflated 42%)
2022-09-27T16:55:56.7907429Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162923.xml (deflated 42%)
2022-09-27T16:55:56.7908240Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162926.xml (deflated 41%)
2022-09-27T16:55:56.7909018Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162936.xml (deflated 41%)
2022-09-27T16:55:56.7909786Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162939.xml (deflated 41%)
2022-09-27T16:55:56.7910543Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162941.xml (deflated 42%)
2022-09-27T16:55:56.7911306Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162944.xml (deflated 41%)
2022-09-27T16:55:56.7912071Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162946.xml (deflated 42%)
2022-09-27T16:55:56.7912842Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162949.xml (deflated 42%)
2022-09-27T16:55:56.7913590Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162956.xml (deflated 42%)
2022-09-27T16:55:56.7914361Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162959.xml (deflated 42%)
2022-09-27T16:55:56.7915125Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163006.xml (deflated 42%)
2022-09-27T16:55:56.7915887Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163009.xml (deflated 42%)
2022-09-27T16:55:56.7916643Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163011.xml (deflated 42%)
2022-09-27T16:55:56.7917403Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163014.xml (deflated 40%)
2022-09-27T16:55:56.7918170Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163016.xml (deflated 40%)
2022-09-27T16:55:56.7918930Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163019.xml (deflated 41%)
2022-09-27T16:55:56.7919741Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163021.xml (deflated 41%)
2022-09-27T16:55:56.7920664Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163024.xml (deflated 40%)
2022-09-27T16:55:56.7921440Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163026.xml (deflated 41%)
2022-09-27T16:55:56.7922210Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163029.xml (deflated 41%)
2022-09-27T16:55:56.7922961Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163031.xml (deflated 40%)
2022-09-27T16:55:56.7923726Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163034.xml (deflated 41%)
2022-09-27T16:55:56.7924490Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163036.xml (deflated 41%)
2022-09-27T16:55:56.7925255Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163038.xml (deflated 40%)
2022-09-27T16:55:56.7926009Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163049.xml (deflated 41%)
2022-09-27T16:55:56.7926771Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163051.xml (deflated 40%)
2022-09-27T16:55:56.7927605Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163101.xml (deflated 42%)
2022-09-27T16:55:56.7928387Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163109.xml (deflated 40%)
2022-09-27T16:55:56.7929136Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163117.xml (deflated 40%)
2022-09-27T16:55:56.7929904Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163120.xml (deflated 40%)
2022-09-27T16:55:56.7930668Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163123.xml (deflated 40%)
2022-09-27T16:55:56.7931425Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163125.xml (deflated 40%)
2022-09-27T16:55:56.7932176Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163135.xml (deflated 40%)
2022-09-27T16:55:56.7932947Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163144.xml (deflated 40%)
2022-09-27T16:55:56.7933707Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163154.xml (deflated 40%)
2022-09-27T16:55:56.7934475Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163204.xml (deflated 40%)
2022-09-27T16:55:56.7935240Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163206.xml (deflated 40%)
2022-09-27T16:55:56.7935993Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163215.xml (deflated 40%)
2022-09-27T16:55:56.7936760Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163224.xml (deflated 40%)
2022-09-27T16:55:56.7937523Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163227.xml (deflated 40%)
2022-09-27T16:55:56.7938282Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163229.xml (deflated 40%)
2022-09-27T16:55:56.7939116Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163238.xml (deflated 40%)
2022-09-27T16:55:56.7939883Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163241.xml (deflated 40%)
2022-09-27T16:55:56.7940638Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163243.xml (deflated 40%)
2022-09-27T16:55:56.7941422Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163253.xml (deflated 40%)
2022-09-27T16:55:56.7942174Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163416.xml (deflated 40%)
2022-09-27T16:55:56.7942942Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163426.xml (deflated 40%)
2022-09-27T16:55:56.7943702Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163436.xml (deflated 39%)
2022-09-27T16:55:56.7944455Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163447.xml (deflated 40%)
2022-09-27T16:55:56.7945200Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163449.xml (deflated 41%)
2022-09-27T16:55:56.7945958Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163452.xml (deflated 41%)
2022-09-27T16:55:56.7946766Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163454.xml (deflated 41%)
2022-09-27T16:55:56.7947542Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163457.xml (deflated 41%)
2022-09-27T16:55:56.7948285Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163459.xml (deflated 41%)
2022-09-27T16:55:56.7949054Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163502.xml (deflated 41%)
2022-09-27T16:55:56.7949815Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163504.xml (deflated 41%)
2022-09-27T16:55:56.7950572Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163507.xml (deflated 41%)
2022-09-27T16:55:56.7951320Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163509.xml (deflated 41%)
2022-09-27T16:55:56.7952091Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163512.xml (deflated 41%)
2022-09-27T16:55:56.7952846Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163515.xml (deflated 41%)
2022-09-27T16:55:56.7953607Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163517.xml (deflated 41%)
2022-09-27T16:55:56.7954353Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163520.xml (deflated 40%)
2022-09-27T16:55:56.7955105Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163522.xml (deflated 40%)
2022-09-27T16:55:56.7955858Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163532.xml (deflated 40%)
2022-09-27T16:55:56.7956626Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163542.xml (deflated 40%)
2022-09-27T16:55:56.7957370Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163552.xml (deflated 40%)
2022-09-27T16:55:56.7958193Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163601.xml (deflated 40%)
2022-09-27T16:55:56.7958957Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163604.xml (deflated 40%)
2022-09-27T16:55:56.7959724Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163606.xml (deflated 41%)
2022-09-27T16:55:56.7960656Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163609.xml (deflated 41%)
2022-09-27T16:55:56.7961107Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163611.xml (deflated 41%)
2022-09-27T16:55:56.7961539Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163614.xml (deflated 40%)
2022-09-27T16:55:56.7961968Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163623.xml (deflated 40%)
2022-09-27T16:55:56.7962400Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163633.xml (deflated 42%)
2022-09-27T16:55:56.7962835Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163636.xml (deflated 42%)
2022-09-27T16:55:56.7963272Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163638.xml (deflated 40%)
2022-09-27T16:55:56.7963781Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163647.xml (deflated 40%)
2022-09-27T16:55:56.7964227Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163657.xml (deflated 40%)
2022-09-27T16:55:56.7964668Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163659.xml (deflated 41%)
2022-09-27T16:55:56.7965087Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163709.xml (deflated 41%)
2022-09-27T16:55:56.7965524Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163720.xml (deflated 41%)
2022-09-27T16:55:56.7965958Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163729.xml (deflated 40%)
2022-09-27T16:55:56.7966394Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163738.xml (deflated 40%)
2022-09-27T16:55:56.7966831Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163748.xml (deflated 42%)
2022-09-27T16:55:56.7967262Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163752.xml (deflated 42%)
2022-09-27T16:55:56.7967702Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163756.xml (deflated 40%)
2022-09-27T16:55:56.7968132Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163759.xml (deflated 40%)
2022-09-27T16:55:56.7968569Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163806.xml (deflated 41%)
2022-09-27T16:55:56.7969003Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163822.xml (deflated 40%)
2022-09-27T16:55:56.7969435Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163832.xml (deflated 41%)
2022-09-27T16:55:56.7969851Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163834.xml (deflated 40%)
2022-09-27T16:55:56.7970364Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163842.xml (deflated 40%)
2022-09-27T16:55:56.7970795Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163844.xml (deflated 40%)
2022-09-27T16:55:56.7971225Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163847.xml (deflated 41%)
2022-09-27T16:55:56.7971662Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163849.xml (deflated 40%)
2022-09-27T16:55:56.7972314Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163852.xml (deflated 40%)
2022-09-27T16:55:56.7972749Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163854.xml (deflated 40%)
2022-09-27T16:55:56.7973189Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163857.xml (deflated 40%)
2022-09-27T16:55:56.7973625Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163900.xml (deflated 41%)
2022-09-27T16:55:56.7974055Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163907.xml (deflated 42%)
2022-09-27T16:55:56.7974534Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163909.xml (deflated 40%)
2022-09-27T16:55:56.7974978Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163917.xml (deflated 40%)
2022-09-27T16:55:56.7975411Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163924.xml (deflated 40%)
2022-09-27T16:55:56.7975849Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163932.xml (deflated 41%)
2022-09-27T16:55:56.7976281Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163934.xml (deflated 40%)
2022-09-27T16:55:56.7976712Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163943.xml (deflated 40%)
2022-09-27T16:55:56.7977146Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163952.xml (deflated 40%)
2022-09-27T16:55:56.7977582Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164003.xml (deflated 41%)
2022-09-27T16:55:56.7978018Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164013.xml (deflated 40%)
2022-09-27T16:55:56.7978451Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164015.xml (deflated 40%)
2022-09-27T16:55:56.7978877Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164018.xml (deflated 42%)
2022-09-27T16:55:56.7979295Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164020.xml (deflated 41%)
2022-09-27T16:55:56.7979722Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164023.xml (deflated 41%)
2022-09-27T16:55:56.7980158Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164025.xml (deflated 41%)
2022-09-27T16:55:56.7980586Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164028.xml (deflated 41%)
2022-09-27T16:55:56.7981015Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164030.xml (deflated 42%)
2022-09-27T16:55:56.7981509Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164033.xml (deflated 41%)
2022-09-27T16:55:56.7981941Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164036.xml (deflated 42%)
2022-09-27T16:55:56.7982373Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164038.xml (deflated 41%)
2022-09-27T16:55:56.7982811Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164041.xml (deflated 42%)
2022-09-27T16:55:56.7983240Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164043.xml (deflated 42%)
2022-09-27T16:55:56.7983656Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164046.xml (deflated 41%)
2022-09-27T16:55:56.7984093Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164049.xml (deflated 41%)
2022-09-27T16:55:56.7984525Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164051.xml (deflated 40%)
2022-09-27T16:55:56.7984953Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164058.xml (deflated 40%)
2022-09-27T16:55:56.7985431Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164105.xml (deflated 41%)
2022-09-27T16:55:56.7985874Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164108.xml (deflated 42%)
2022-09-27T16:55:56.7986306Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164110.xml (deflated 42%)
2022-09-27T16:55:56.7986738Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164115.xml (deflated 40%)
2022-09-27T16:55:56.7987165Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164124.xml (deflated 40%)
2022-09-27T16:55:56.7987597Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164135.xml (deflated 40%)
2022-09-27T16:55:56.7988033Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164145.xml (deflated 40%)
2022-09-27T16:55:56.7988453Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164155.xml (deflated 39%)
2022-09-27T16:55:56.7988884Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164205.xml (deflated 40%)
2022-09-27T16:55:56.7989320Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164212.xml (deflated 40%)
2022-09-27T16:55:56.7989749Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164219.xml (deflated 39%)
2022-09-27T16:55:56.7990181Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164229.xml (deflated 40%)
2022-09-27T16:55:56.7990611Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164239.xml (deflated 40%)
2022-09-27T16:55:56.7991047Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164249.xml (deflated 40%)
2022-09-27T16:55:56.7991477Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164300.xml (deflated 40%)
2022-09-27T16:55:56.7991973Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164310.xml (deflated 42%)
2022-09-27T16:55:56.7992405Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164317.xml (deflated 42%)
2022-09-27T16:55:56.7992822Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164324.xml (deflated 42%)
2022-09-27T16:55:56.7993260Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164331.xml (deflated 42%)
2022-09-27T16:55:56.7993697Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164338.xml (deflated 40%)
2022-09-27T16:55:56.7994128Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164348.xml (deflated 41%)
2022-09-27T16:55:56.7994557Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164358.xml (deflated 42%)
2022-09-27T16:55:56.7994986Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164401.xml (deflated 41%)
2022-09-27T16:55:56.7995411Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164411.xml (deflated 43%)
2022-09-27T16:55:56.7995843Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164413.xml (deflated 40%)
2022-09-27T16:55:56.7996314Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164424.xml (deflated 42%)
2022-09-27T16:55:56.7996757Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164426.xml (deflated 42%)
2022-09-27T16:55:56.7997186Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164429.xml (deflated 40%)
2022-09-27T16:55:56.7997607Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164439.xml (deflated 41%)
2022-09-27T16:55:56.7998041Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164441.xml (deflated 41%)
2022-09-27T16:55:56.7998476Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164444.xml (deflated 41%)
2022-09-27T16:55:56.7998911Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164446.xml (deflated 41%)
2022-09-27T16:55:56.7999344Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164449.xml (deflated 41%)
2022-09-27T16:55:56.7999773Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164452.xml (deflated 41%)
2022-09-27T16:55:56.8000366Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164454.xml (deflated 40%)
2022-09-27T16:55:56.8000806Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164457.xml (deflated 41%)
2022-09-27T16:55:56.8001241Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164459.xml (deflated 40%)
2022-09-27T16:55:56.8001676Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164508.xml (deflated 42%)
2022-09-27T16:55:56.8002092Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164511.xml (deflated 42%)
2022-09-27T16:55:56.8002522Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164513.xml (deflated 42%)
2022-09-27T16:55:56.8003116Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164516.xml (deflated 40%)
2022-09-27T16:55:56.8003543Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164526.xml (deflated 41%)
2022-09-27T16:55:56.8003967Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164528.xml (deflated 40%)
2022-09-27T16:55:56.8004396Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164531.xml (deflated 40%)
2022-09-27T16:55:56.8004826Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164534.xml (deflated 40%)
2022-09-27T16:55:56.8005258Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164544.xml (deflated 41%)
2022-09-27T16:55:56.8005689Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164553.xml (deflated 40%)
2022-09-27T16:55:56.8006118Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164603.xml (deflated 40%)
2022-09-27T16:55:56.8006545Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164613.xml (deflated 42%)
2022-09-27T16:55:56.8006963Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164616.xml (deflated 42%)
2022-09-27T16:55:56.8007455Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164619.xml (deflated 41%)
2022-09-27T16:55:56.8007902Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164628.xml (deflated 40%)
2022-09-27T16:55:56.8008328Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164637.xml (deflated 41%)
2022-09-27T16:55:56.8008765Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164640.xml (deflated 41%)
2022-09-27T16:55:56.8009194Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164657.xml (deflated 40%)
2022-09-27T16:55:56.8009620Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164659.xml (deflated 40%)
2022-09-27T16:55:56.8010065Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154655.xml (deflated 41%)
2022-09-27T16:55:56.8010503Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154706.xml (deflated 41%)
2022-09-27T16:55:56.8010937Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154708.xml (deflated 41%)
2022-09-27T16:55:56.8011363Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154715.xml (deflated 64%)
2022-09-27T16:55:56.8011798Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154841.xml (deflated 42%)
2022-09-27T16:55:56.8012231Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154843.xml (deflated 41%)
2022-09-27T16:55:56.8012670Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154846.xml (deflated 40%)
2022-09-27T16:55:56.8013101Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154857.xml (deflated 40%)
2022-09-27T16:55:56.8013531Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154907.xml (deflated 40%)
2022-09-27T16:55:56.8014029Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154917.xml (deflated 39%)
2022-09-27T16:55:56.8014463Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154927.xml (deflated 39%)
2022-09-27T16:55:56.8014897Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154929.xml (deflated 40%)
2022-09-27T16:55:56.8015330Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154940.xml (deflated 40%)
2022-09-27T16:55:56.8015761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154950.xml (deflated 42%)
2022-09-27T16:55:56.8016180Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927154954.xml (deflated 41%)
2022-09-27T16:55:56.8016616Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155002.xml (deflated 41%)
2022-09-27T16:55:56.8017046Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155011.xml (deflated 41%)
2022-09-27T16:55:56.8017480Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155021.xml (deflated 42%)
2022-09-27T16:55:56.8017908Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155023.xml (deflated 46%)
2022-09-27T16:55:56.8018383Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155034.xml (deflated 47%)
2022-09-27T16:55:56.8018824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155044.xml (deflated 48%)
2022-09-27T16:55:56.8019261Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155054.xml (deflated 45%)
2022-09-27T16:55:56.8019692Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155104.xml (deflated 41%)
2022-09-27T16:55:56.8020123Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155107.xml (deflated 43%)
2022-09-27T16:55:56.8020537Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155110.xml (deflated 43%)
2022-09-27T16:55:56.8020975Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155112.xml (deflated 43%)
2022-09-27T16:55:56.8021408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155115.xml (deflated 43%)
2022-09-27T16:55:56.8021839Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155117.xml (deflated 43%)
2022-09-27T16:55:56.8022274Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155120.xml (deflated 41%)
2022-09-27T16:55:56.8022702Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155122.xml (deflated 40%)
2022-09-27T16:55:56.8023133Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155133.xml (deflated 40%)
2022-09-27T16:55:56.8023564Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155144.xml (deflated 41%)
2022-09-27T16:55:56.8023988Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155146.xml (deflated 41%)
2022-09-27T16:55:56.8024417Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155149.xml (deflated 42%)
2022-09-27T16:55:56.8024901Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155153.xml (deflated 42%)
2022-09-27T16:55:56.8025333Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155157.xml (deflated 40%)
2022-09-27T16:55:56.8025759Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155207.xml (deflated 40%)
2022-09-27T16:55:56.8026192Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155218.xml (deflated 41%)
2022-09-27T16:55:56.8026620Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155228.xml (deflated 41%)
2022-09-27T16:55:56.8027051Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155231.xml (deflated 41%)
2022-09-27T16:55:56.8027484Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155233.xml (deflated 41%)
2022-09-27T16:55:56.8027914Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155236.xml (deflated 41%)
2022-09-27T16:55:56.8028343Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155238.xml (deflated 40%)
2022-09-27T16:55:56.8028820Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155241.xml (deflated 41%)
2022-09-27T16:55:56.8029262Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155243.xml (deflated 41%)
2022-09-27T16:55:56.8029675Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155246.xml (deflated 41%)
2022-09-27T16:55:56.8030116Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155248.xml (deflated 41%)
2022-09-27T16:55:56.8030547Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155251.xml (deflated 41%)
2022-09-27T16:55:56.8030977Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155253.xml (deflated 41%)
2022-09-27T16:55:56.8031407Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155256.xml (deflated 41%)
2022-09-27T16:55:56.8031835Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155259.xml (deflated 41%)
2022-09-27T16:55:56.8032259Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155301.xml (deflated 41%)
2022-09-27T16:55:56.8032687Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155304.xml (deflated 40%)
2022-09-27T16:55:56.8033112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155306.xml (deflated 41%)
2022-09-27T16:55:56.8033541Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155309.xml (deflated 41%)
2022-09-27T16:55:56.8033953Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155311.xml (deflated 41%)
2022-09-27T16:55:56.8034385Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155314.xml (deflated 41%)
2022-09-27T16:55:56.8034815Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155317.xml (deflated 41%)
2022-09-27T16:55:56.8035309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155319.xml (deflated 41%)
2022-09-27T16:55:56.8035738Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155322.xml (deflated 41%)
2022-09-27T16:55:56.8036170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155324.xml (deflated 41%)
2022-09-27T16:55:56.8036598Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155327.xml (deflated 41%)
2022-09-27T16:55:56.8037030Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155329.xml (deflated 43%)
2022-09-27T16:55:56.8037462Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155332.xml (deflated 42%)
2022-09-27T16:55:56.8037890Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155335.xml (deflated 41%)
2022-09-27T16:55:56.8038555Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155337.xml (deflated 40%)
2022-09-27T16:55:56.8038997Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155347.xml (deflated 41%)
2022-09-27T16:55:56.8039427Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155349.xml (deflated 41%)
2022-09-27T16:55:56.8039919Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155352.xml (deflated 41%)
2022-09-27T16:55:56.8040518Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155355.xml (deflated 40%)
2022-09-27T16:55:56.8040958Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155404.xml (deflated 40%)
2022-09-27T16:55:56.8041392Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155413.xml (deflated 41%)
2022-09-27T16:55:56.8041821Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155422.xml (deflated 42%)
2022-09-27T16:55:56.8042253Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155425.xml (deflated 42%)
2022-09-27T16:55:56.8042685Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155427.xml (deflated 41%)
2022-09-27T16:55:56.8043116Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155437.xml (deflated 40%)
2022-09-27T16:55:56.8043530Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155447.xml (deflated 42%)
2022-09-27T16:55:56.8043966Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155450.xml (deflated 41%)
2022-09-27T16:55:56.8044398Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155500.xml (deflated 42%)
2022-09-27T16:55:56.8044829Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155502.xml (deflated 42%)
2022-09-27T16:55:56.8045260Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155509.xml (deflated 42%)
2022-09-27T16:55:56.8045690Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155512.xml (deflated 42%)
2022-09-27T16:55:56.8046119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155514.xml (deflated 42%)
2022-09-27T16:55:56.8046645Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155519.xml (deflated 41%)
2022-09-27T16:55:56.8047075Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155530.xml (deflated 42%)
2022-09-27T16:55:56.8047503Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155533.xml (deflated 42%)
2022-09-27T16:55:56.8047935Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155537.xml (deflated 42%)
2022-09-27T16:55:56.8048350Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155540.xml (deflated 42%)
2022-09-27T16:55:56.8048779Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155546.xml (deflated 42%)
2022-09-27T16:55:56.8049211Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155549.xml (deflated 42%)
2022-09-27T16:55:56.8049640Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155551.xml (deflated 41%)
2022-09-27T16:55:56.8050067Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155601.xml (deflated 40%)
2022-09-27T16:55:56.8050495Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155611.xml (deflated 42%)
2022-09-27T16:55:56.8050993Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155613.xml (deflated 40%)
2022-09-27T16:55:56.8051436Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155623.xml (deflated 42%)
2022-09-27T16:55:56.8051869Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155626.xml (deflated 41%)
2022-09-27T16:55:56.8052299Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155633.xml (deflated 40%)
2022-09-27T16:55:56.8052714Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155643.xml (deflated 41%)
2022-09-27T16:55:56.8053144Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155650.xml (deflated 42%)
2022-09-27T16:55:56.8053577Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155652.xml (deflated 42%)
2022-09-27T16:55:56.8054008Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155655.xml (deflated 40%)
2022-09-27T16:55:56.8054438Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155705.xml (deflated 42%)
2022-09-27T16:55:56.8054865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155708.xml (deflated 42%)
2022-09-27T16:55:56.8055294Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155715.xml (deflated 42%)
2022-09-27T16:55:56.8055724Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155718.xml (deflated 42%)
2022-09-27T16:55:56.8056156Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155725.xml (deflated 42%)
2022-09-27T16:55:56.8056586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155727.xml (deflated 43%)
2022-09-27T16:55:56.8057015Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155730.xml (deflated 42%)
2022-09-27T16:55:56.8057492Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155732.xml (deflated 40%)
2022-09-27T16:55:56.8057925Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155735.xml (deflated 40%)
2022-09-27T16:55:56.8058354Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155737.xml (deflated 41%)
2022-09-27T16:55:56.8058786Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155744.xml (deflated 40%)
2022-09-27T16:55:56.8059218Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155753.xml (deflated 41%)
2022-09-27T16:55:56.8059649Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155801.xml (deflated 41%)
2022-09-27T16:55:56.8060115Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155809.xml (deflated 40%)
2022-09-27T16:55:56.8060547Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155816.xml (deflated 40%)
2022-09-27T16:55:56.8060977Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155825.xml (deflated 41%)
2022-09-27T16:55:56.8061452Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155834.xml (deflated 41%)
2022-09-27T16:55:56.8061877Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155841.xml (deflated 41%)
2022-09-27T16:55:56.8062308Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155844.xml (deflated 42%)
2022-09-27T16:55:56.8062742Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155848.xml (deflated 41%)
2022-09-27T16:55:56.8063170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155850.xml (deflated 42%)
2022-09-27T16:55:56.8063596Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155853.xml (deflated 43%)
2022-09-27T16:55:56.8064026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155855.xml (deflated 42%)
2022-09-27T16:55:56.8064457Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155900.xml (deflated 42%)
2022-09-27T16:55:56.8064885Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155904.xml (deflated 42%)
2022-09-27T16:55:56.8065317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155909.xml (deflated 40%)
2022-09-27T16:55:56.8065747Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155918.xml (deflated 40%)
2022-09-27T16:55:56.8066161Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155927.xml (deflated 40%)
2022-09-27T16:55:56.8066592Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155936.xml (deflated 40%)
2022-09-27T16:55:56.8067023Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155946.xml (deflated 40%)
2022-09-27T16:55:56.8067452Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927155955.xml (deflated 40%)
2022-09-27T16:55:56.8067992Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160005.xml (deflated 40%)
2022-09-27T16:55:56.8068423Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160015.xml (deflated 40%)
2022-09-27T16:55:56.8068851Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160025.xml (deflated 41%)
2022-09-27T16:55:56.8069281Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160034.xml (deflated 42%)
2022-09-27T16:55:56.8069715Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160036.xml (deflated 41%)
2022-09-27T16:55:56.8070143Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160047.xml (deflated 41%)
2022-09-27T16:55:56.8070570Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160057.xml (deflated 40%)
2022-09-27T16:55:56.8070988Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160108.xml (deflated 40%)
2022-09-27T16:55:56.8071418Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160118.xml (deflated 40%)
2022-09-27T16:55:56.8071849Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160129.xml (deflated 40%)
2022-09-27T16:55:56.8072328Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160140.xml (deflated 40%)
2022-09-27T16:55:56.8072767Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160152.xml (deflated 40%)
2022-09-27T16:55:56.8073194Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160202.xml (deflated 42%)
2022-09-27T16:55:56.8073628Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160205.xml (deflated 42%)
2022-09-27T16:55:56.8074057Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160208.xml (deflated 42%)
2022-09-27T16:55:56.8074488Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160210.xml (deflated 41%)
2022-09-27T16:55:56.8074918Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160213.xml (deflated 41%)
2022-09-27T16:55:56.8075335Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160215.xml (deflated 41%)
2022-09-27T16:55:56.8075766Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160218.xml (deflated 41%)
2022-09-27T16:55:56.8076202Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160220.xml (deflated 41%)
2022-09-27T16:55:56.8076633Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160223.xml (deflated 41%)
2022-09-27T16:55:56.8077065Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160225.xml (deflated 41%)
2022-09-27T16:55:56.8077495Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160228.xml (deflated 42%)
2022-09-27T16:55:56.8077926Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160230.xml (deflated 42%)
2022-09-27T16:55:56.8078354Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160233.xml (deflated 41%)
2022-09-27T16:55:56.8078862Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160243.xml (deflated 40%)
2022-09-27T16:55:56.8079292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160253.xml (deflated 40%)
2022-09-27T16:55:56.8079720Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160304.xml (deflated 41%)
2022-09-27T16:55:56.8080138Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160306.xml (deflated 40%)
2022-09-27T16:55:56.8080736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160317.xml (deflated 40%)
2022-09-27T16:55:56.8081168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160326.xml (deflated 40%)
2022-09-27T16:55:56.8081602Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160345.xml (deflated 41%)
2022-09-27T16:55:56.8082030Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160355.xml (deflated 41%)
2022-09-27T16:55:56.8082461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160407.xml (deflated 41%)
2022-09-27T16:55:56.8082886Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160417.xml (deflated 40%)
2022-09-27T16:55:56.8083379Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160428.xml (deflated 40%)
2022-09-27T16:55:56.8083822Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160438.xml (deflated 41%)
2022-09-27T16:55:56.8084253Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160448.xml (deflated 41%)
2022-09-27T16:55:56.8084667Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160458.xml (deflated 40%)
2022-09-27T16:55:56.8085097Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160507.xml (deflated 40%)
2022-09-27T16:55:56.8085527Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160517.xml (deflated 40%)
2022-09-27T16:55:56.8085959Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160526.xml (deflated 41%)
2022-09-27T16:55:56.8086388Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160535.xml (deflated 41%)
2022-09-27T16:55:56.8086814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160545.xml (deflated 41%)
2022-09-27T16:55:56.8087243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160555.xml (deflated 40%)
2022-09-27T16:55:56.8087672Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160605.xml (deflated 40%)
2022-09-27T16:55:56.8088098Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160616.xml (deflated 42%)
2022-09-27T16:55:56.8088529Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160620.xml (deflated 42%)
2022-09-27T16:55:56.8088959Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160624.xml (deflated 40%)
2022-09-27T16:55:56.8089374Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160634.xml (deflated 40%)
2022-09-27T16:55:56.8089888Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160642.xml (deflated 40%)
2022-09-27T16:55:56.8090320Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160649.xml (deflated 40%)
2022-09-27T16:55:56.8090750Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160658.xml (deflated 41%)
2022-09-27T16:55:56.8091183Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160708.xml (deflated 40%)
2022-09-27T16:55:56.8091613Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160715.xml (deflated 41%)
2022-09-27T16:55:56.8092040Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160718.xml (deflated 41%)
2022-09-27T16:55:56.8092473Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160721.xml (deflated 41%)
2022-09-27T16:55:56.8092902Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160732.xml (deflated 41%)
2022-09-27T16:55:56.8093333Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160734.xml (deflated 41%)
2022-09-27T16:55:56.8093793Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160737.xml (deflated 40%)
2022-09-27T16:55:56.8094232Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160747.xml (deflated 42%)
2022-09-27T16:55:56.8094662Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160751.xml (deflated 41%)
2022-09-27T16:55:56.8095091Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160758.xml (deflated 41%)
2022-09-27T16:55:56.8095516Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160812.xml (deflated 40%)
2022-09-27T16:55:56.8095946Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160819.xml (deflated 40%)
2022-09-27T16:55:56.8096375Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160827.xml (deflated 40%)
2022-09-27T16:55:56.8096806Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160834.xml (deflated 40%)
2022-09-27T16:55:56.8097239Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160843.xml (deflated 42%)
2022-09-27T16:55:56.8097670Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160846.xml (deflated 42%)
2022-09-27T16:55:56.8098083Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160848.xml (deflated 42%)
2022-09-27T16:55:56.8098516Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160851.xml (deflated 42%)
2022-09-27T16:55:56.8098946Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160854.xml (deflated 40%)
2022-09-27T16:55:56.8099380Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160919.xml (deflated 40%)
2022-09-27T16:55:56.8099809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160945.xml (deflated 42%)
2022-09-27T16:55:56.8100309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160947.xml (deflated 41%)
2022-09-27T16:55:56.8100740Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160950.xml (deflated 41%)
2022-09-27T16:55:56.8101169Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160952.xml (deflated 41%)
2022-09-27T16:55:56.8101597Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160955.xml (deflated 41%)
2022-09-27T16:55:56.8102029Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927160957.xml (deflated 41%)
2022-09-27T16:55:56.8102455Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161006.xml (deflated 41%)
2022-09-27T16:55:56.8102870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161015.xml (deflated 41%)
2022-09-27T16:55:56.8103304Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161024.xml (deflated 41%)
2022-09-27T16:55:56.8103728Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161033.xml (deflated 42%)
2022-09-27T16:55:56.8104156Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161037.xml (deflated 42%)
2022-09-27T16:55:56.8104628Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161040.xml (deflated 42%)
2022-09-27T16:55:56.8105067Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161042.xml (deflated 41%)
2022-09-27T16:55:56.8105496Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161045.xml (deflated 40%)
2022-09-27T16:55:56.8105927Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161052.xml (deflated 40%)
2022-09-27T16:55:56.8106355Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161059.xml (deflated 41%)
2022-09-27T16:55:56.8106786Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161102.xml (deflated 42%)
2022-09-27T16:55:56.8107202Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161104.xml (deflated 42%)
2022-09-27T16:55:56.8107635Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161109.xml (deflated 40%)
2022-09-27T16:55:56.8108066Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161129.xml (deflated 41%)
2022-09-27T16:55:56.8108495Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161139.xml (deflated 40%)
2022-09-27T16:55:56.8108924Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161149.xml (deflated 40%)
2022-09-27T16:55:56.8109355Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161159.xml (deflated 39%)
2022-09-27T16:55:56.8109785Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161208.xml (deflated 40%)
2022-09-27T16:55:56.8110214Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161215.xml (deflated 40%)
2022-09-27T16:55:56.8110637Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161222.xml (deflated 40%)
2022-09-27T16:55:56.8111126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161237.xml (deflated 41%)
2022-09-27T16:55:56.8111558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161240.xml (deflated 41%)
2022-09-27T16:55:56.8111972Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161242.xml (deflated 41%)
2022-09-27T16:55:56.8112404Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161245.xml (deflated 41%)
2022-09-27T16:55:56.8112834Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161247.xml (deflated 41%)
2022-09-27T16:55:56.8113260Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161249.xml (deflated 41%)
2022-09-27T16:55:56.8113686Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161252.xml (deflated 41%)
2022-09-27T16:55:56.8114114Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161254.xml (deflated 41%)
2022-09-27T16:55:56.8114542Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161256.xml (deflated 41%)
2022-09-27T16:55:56.8114970Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161259.xml (deflated 41%)
2022-09-27T16:55:56.8115446Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161301.xml (deflated 40%)
2022-09-27T16:55:56.8115887Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161310.xml (deflated 41%)
2022-09-27T16:55:56.8116306Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161312.xml (deflated 40%)
2022-09-27T16:55:56.8116734Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161322.xml (deflated 41%)
2022-09-27T16:55:56.8117161Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161324.xml (deflated 40%)
2022-09-27T16:55:56.8117591Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161333.xml (deflated 40%)
2022-09-27T16:55:56.8118024Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161342.xml (deflated 41%)
2022-09-27T16:55:56.8118451Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161345.xml (deflated 41%)
2022-09-27T16:55:56.8118879Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161347.xml (deflated 41%)
2022-09-27T16:55:56.8119311Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161349.xml (deflated 41%)
2022-09-27T16:55:56.8119740Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161352.xml (deflated 40%)
2022-09-27T16:55:56.8120313Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161402.xml (deflated 40%)
2022-09-27T16:55:56.8120760Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161413.xml (deflated 41%)
2022-09-27T16:55:56.8121176Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161415.xml (deflated 41%)
2022-09-27T16:55:56.8121605Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161418.xml (deflated 41%)
2022-09-27T16:55:56.8122119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161420.xml (deflated 43%)
2022-09-27T16:55:56.8122548Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161423.xml (deflated 42%)
2022-09-27T16:55:56.8122976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161425.xml (deflated 42%)
2022-09-27T16:55:56.8123408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161427.xml (deflated 42%)
2022-09-27T16:55:56.8123836Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161430.xml (deflated 43%)
2022-09-27T16:55:56.8124263Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161432.xml (deflated 40%)
2022-09-27T16:55:56.8124692Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161441.xml (deflated 40%)
2022-09-27T16:55:56.8125117Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161450.xml (deflated 40%)
2022-09-27T16:55:56.8125531Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161458.xml (deflated 42%)
2022-09-27T16:55:56.8126014Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161501.xml (deflated 43%)
2022-09-27T16:55:56.8126456Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161503.xml (deflated 42%)
2022-09-27T16:55:56.8126885Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161506.xml (deflated 42%)
2022-09-27T16:55:56.8127316Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161508.xml (deflated 42%)
2022-09-27T16:55:56.8127746Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161511.xml (deflated 42%)
2022-09-27T16:55:56.8128173Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161513.xml (deflated 41%)
2022-09-27T16:55:56.8128605Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161523.xml (deflated 41%)
2022-09-27T16:55:56.8129034Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161526.xml (deflated 40%)
2022-09-27T16:55:56.8129460Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161535.xml (deflated 41%)
2022-09-27T16:55:56.8129881Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161545.xml (deflated 40%)
2022-09-27T16:55:56.8130310Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927161559.xml (deflated 40%)
2022-09-27T16:55:56.8130733Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162013.xml (deflated 41%)
2022-09-27T16:55:56.8131157Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162023.xml (deflated 41%)
2022-09-27T16:55:56.8131586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162025.xml (deflated 42%)
2022-09-27T16:55:56.8132015Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162033.xml (deflated 64%)
2022-09-27T16:55:56.8132508Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162158.xml (deflated 42%)
2022-09-27T16:55:56.8132939Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162201.xml (deflated 42%)
2022-09-27T16:55:56.8133370Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162203.xml (deflated 41%)
2022-09-27T16:55:56.8133800Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162214.xml (deflated 40%)
2022-09-27T16:55:56.8134233Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162224.xml (deflated 40%)
2022-09-27T16:55:56.8134647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162234.xml (deflated 39%)
2022-09-27T16:55:56.8135074Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162244.xml (deflated 40%)
2022-09-27T16:55:56.8135507Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162247.xml (deflated 40%)
2022-09-27T16:55:56.8135937Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162257.xml (deflated 40%)
2022-09-27T16:55:56.8136364Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162307.xml (deflated 42%)
2022-09-27T16:55:56.8136838Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162312.xml (deflated 41%)
2022-09-27T16:55:56.8137276Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162319.xml (deflated 41%)
2022-09-27T16:55:56.8137700Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162328.xml (deflated 41%)
2022-09-27T16:55:56.8138129Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162338.xml (deflated 41%)
2022-09-27T16:55:56.8138555Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162340.xml (deflated 45%)
2022-09-27T16:55:56.8138969Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162351.xml (deflated 47%)
2022-09-27T16:55:56.8139397Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162401.xml (deflated 48%)
2022-09-27T16:55:56.8139825Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162411.xml (deflated 45%)
2022-09-27T16:55:56.8140257Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162422.xml (deflated 41%)
2022-09-27T16:55:56.8140691Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162425.xml (deflated 43%)
2022-09-27T16:55:56.8141120Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162427.xml (deflated 43%)
2022-09-27T16:55:56.8141543Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162429.xml (deflated 43%)
2022-09-27T16:55:56.8141973Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162432.xml (deflated 43%)
2022-09-27T16:55:56.8142399Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162434.xml (deflated 43%)
2022-09-27T16:55:56.8142827Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162437.xml (deflated 41%)
2022-09-27T16:55:56.8143320Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162440.xml (deflated 40%)
2022-09-27T16:55:56.8143736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162451.xml (deflated 40%)
2022-09-27T16:55:56.8144166Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162502.xml (deflated 41%)
2022-09-27T16:55:56.8144596Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162505.xml (deflated 41%)
2022-09-27T16:55:56.8145024Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162507.xml (deflated 42%)
2022-09-27T16:55:56.8145453Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162512.xml (deflated 42%)
2022-09-27T16:55:56.8145887Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162516.xml (deflated 40%)
2022-09-27T16:55:56.8146317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162526.xml (deflated 40%)
2022-09-27T16:55:56.8146748Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162537.xml (deflated 40%)
2022-09-27T16:55:56.8147176Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162547.xml (deflated 41%)
2022-09-27T16:55:56.8147650Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162549.xml (deflated 41%)
2022-09-27T16:55:56.8148075Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162552.xml (deflated 41%)
2022-09-27T16:55:56.8148508Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162554.xml (deflated 41%)
2022-09-27T16:55:56.8148941Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162557.xml (deflated 40%)
2022-09-27T16:55:56.8149370Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162559.xml (deflated 40%)
2022-09-27T16:55:56.8149797Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162602.xml (deflated 41%)
2022-09-27T16:55:56.8150233Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162604.xml (deflated 41%)
2022-09-27T16:55:56.8150666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162607.xml (deflated 41%)
2022-09-27T16:55:56.8151094Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162609.xml (deflated 41%)
2022-09-27T16:55:56.8151524Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162612.xml (deflated 41%)
2022-09-27T16:55:56.8151954Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162614.xml (deflated 41%)
2022-09-27T16:55:56.8152369Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162617.xml (deflated 41%)
2022-09-27T16:55:56.8152801Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162619.xml (deflated 41%)
2022-09-27T16:55:56.8153234Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162622.xml (deflated 41%)
2022-09-27T16:55:56.8153661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162624.xml (deflated 41%)
2022-09-27T16:55:56.8154160Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162627.xml (deflated 41%)
2022-09-27T16:55:56.8154590Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162629.xml (deflated 41%)
2022-09-27T16:55:56.8155018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162632.xml (deflated 41%)
2022-09-27T16:55:56.8155455Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162635.xml (deflated 41%)
2022-09-27T16:55:56.8155883Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162637.xml (deflated 41%)
2022-09-27T16:55:56.8156312Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162640.xml (deflated 41%)
2022-09-27T16:55:56.8156743Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162642.xml (deflated 41%)
2022-09-27T16:55:56.8157157Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162645.xml (deflated 41%)
2022-09-27T16:55:56.8157584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162647.xml (deflated 43%)
2022-09-27T16:55:56.8158056Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162650.xml (deflated 42%)
2022-09-27T16:55:56.8158492Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162652.xml (deflated 41%)
2022-09-27T16:55:56.8158921Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162655.xml (deflated 41%)
2022-09-27T16:55:56.8159350Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162705.xml (deflated 41%)
2022-09-27T16:55:56.8159777Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162707.xml (deflated 41%)
2022-09-27T16:55:56.8160237Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162710.xml (deflated 41%)
2022-09-27T16:55:56.8160797Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162712.xml (deflated 40%)
2022-09-27T16:55:56.8161238Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162722.xml (deflated 40%)
2022-09-27T16:55:56.8161652Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162731.xml (deflated 41%)
2022-09-27T16:55:56.8162084Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162740.xml (deflated 42%)
2022-09-27T16:55:56.8162511Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162743.xml (deflated 42%)
2022-09-27T16:55:56.8162934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162745.xml (deflated 40%)
2022-09-27T16:55:56.8163362Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162755.xml (deflated 40%)
2022-09-27T16:55:56.8163789Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162805.xml (deflated 42%)
2022-09-27T16:55:56.8164215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162808.xml (deflated 41%)
2022-09-27T16:55:56.8164727Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162818.xml (deflated 42%)
2022-09-27T16:55:56.8165158Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162820.xml (deflated 42%)
2022-09-27T16:55:56.8165586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162827.xml (deflated 42%)
2022-09-27T16:55:56.8166018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162830.xml (deflated 42%)
2022-09-27T16:55:56.8166435Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162833.xml (deflated 42%)
2022-09-27T16:55:56.8166864Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162837.xml (deflated 41%)
2022-09-27T16:55:56.8167291Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162848.xml (deflated 42%)
2022-09-27T16:55:56.8167719Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162850.xml (deflated 42%)
2022-09-27T16:55:56.8168146Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162854.xml (deflated 42%)
2022-09-27T16:55:56.8168573Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162857.xml (deflated 42%)
2022-09-27T16:55:56.8169065Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162904.xml (deflated 42%)
2022-09-27T16:55:56.8169507Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162906.xml (deflated 42%)
2022-09-27T16:55:56.8169934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162909.xml (deflated 41%)
2022-09-27T16:55:56.8170367Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162918.xml (deflated 40%)
2022-09-27T16:55:56.8170779Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162928.xml (deflated 42%)
2022-09-27T16:55:56.8171206Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162931.xml (deflated 40%)
2022-09-27T16:55:56.8171638Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162941.xml (deflated 42%)
2022-09-27T16:55:56.8172066Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162943.xml (deflated 41%)
2022-09-27T16:55:56.8172492Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927162950.xml (deflated 41%)
2022-09-27T16:55:56.8172924Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163000.xml (deflated 41%)
2022-09-27T16:55:56.8173347Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163007.xml (deflated 42%)
2022-09-27T16:55:56.8173774Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163010.xml (deflated 42%)
2022-09-27T16:55:56.8174205Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163012.xml (deflated 40%)
2022-09-27T16:55:56.8174631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163022.xml (deflated 42%)
2022-09-27T16:55:56.8175057Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163025.xml (deflated 42%)
2022-09-27T16:55:56.8175533Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163032.xml (deflated 42%)
2022-09-27T16:55:56.8175956Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163034.xml (deflated 42%)
2022-09-27T16:55:56.8176385Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163041.xml (deflated 42%)
2022-09-27T16:55:56.8176813Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163044.xml (deflated 42%)
2022-09-27T16:55:56.8177244Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163046.xml (deflated 42%)
2022-09-27T16:55:56.8177671Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163049.xml (deflated 40%)
2022-09-27T16:55:56.8178102Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163052.xml (deflated 40%)
2022-09-27T16:55:56.8178530Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163054.xml (deflated 41%)
2022-09-27T16:55:56.8178960Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163101.xml (deflated 40%)
2022-09-27T16:55:56.8179387Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163110.xml (deflated 41%)
2022-09-27T16:55:56.8179841Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163117.xml (deflated 41%)
2022-09-27T16:55:56.8180280Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163126.xml (deflated 40%)
2022-09-27T16:55:56.8180711Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163133.xml (deflated 40%)
2022-09-27T16:55:56.8181137Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163142.xml (deflated 40%)
2022-09-27T16:55:56.8181559Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163150.xml (deflated 41%)
2022-09-27T16:55:56.8181984Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163158.xml (deflated 41%)
2022-09-27T16:55:56.8182414Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163200.xml (deflated 42%)
2022-09-27T16:55:56.8182842Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163205.xml (deflated 41%)
2022-09-27T16:55:56.8183268Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163207.xml (deflated 41%)
2022-09-27T16:55:56.8183697Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163210.xml (deflated 43%)
2022-09-27T16:55:56.8184109Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163212.xml (deflated 42%)
2022-09-27T16:55:56.8184533Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163217.xml (deflated 42%)
2022-09-27T16:55:56.8184963Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163221.xml (deflated 42%)
2022-09-27T16:55:56.8185393Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163226.xml (deflated 40%)
2022-09-27T16:55:56.8185823Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163235.xml (deflated 40%)
2022-09-27T16:55:56.8186302Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163244.xml (deflated 40%)
2022-09-27T16:55:56.8186733Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163253.xml (deflated 64%)
2022-09-27T16:55:56.8187159Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163418.xml (deflated 40%)
2022-09-27T16:55:56.8187592Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163428.xml (deflated 40%)
2022-09-27T16:55:56.8188021Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163438.xml (deflated 40%)
2022-09-27T16:55:56.8188449Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163449.xml (deflated 40%)
2022-09-27T16:55:56.8188865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163458.xml (deflated 40%)
2022-09-27T16:55:56.8189292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163507.xml (deflated 42%)
2022-09-27T16:55:56.8189721Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163510.xml (deflated 41%)
2022-09-27T16:55:56.8190191Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163520.xml (deflated 41%)
2022-09-27T16:55:56.8190629Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163530.xml (deflated 40%)
2022-09-27T16:55:56.8191054Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163541.xml (deflated 40%)
2022-09-27T16:55:56.8191484Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163552.xml (deflated 39%)
2022-09-27T16:55:56.8191915Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163603.xml (deflated 40%)
2022-09-27T16:55:56.8192342Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163614.xml (deflated 39%)
2022-09-27T16:55:56.8192766Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163625.xml (deflated 40%)
2022-09-27T16:55:56.8193181Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163635.xml (deflated 41%)
2022-09-27T16:55:56.8193607Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163638.xml (deflated 41%)
2022-09-27T16:55:56.8194029Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163640.xml (deflated 41%)
2022-09-27T16:55:56.8194457Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163643.xml (deflated 41%)
2022-09-27T16:55:56.8194882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163646.xml (deflated 41%)
2022-09-27T16:55:56.8195305Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163648.xml (deflated 41%)
2022-09-27T16:55:56.8195734Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163650.xml (deflated 41%)
2022-09-27T16:55:56.8196174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163653.xml (deflated 41%)
2022-09-27T16:55:56.8196675Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163656.xml (deflated 41%)
2022-09-27T16:55:56.8197112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163658.xml (deflated 41%)
2022-09-27T16:55:56.8197544Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163701.xml (deflated 41%)
2022-09-27T16:55:56.8197965Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163703.xml (deflated 42%)
2022-09-27T16:55:56.8198394Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163706.xml (deflated 40%)
2022-09-27T16:55:56.8198822Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163716.xml (deflated 41%)
2022-09-27T16:55:56.8199249Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163726.xml (deflated 40%)
2022-09-27T16:55:56.8199678Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163737.xml (deflated 41%)
2022-09-27T16:55:56.8200107Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163739.xml (deflated 40%)
2022-09-27T16:55:56.8200691Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163750.xml (deflated 40%)
2022-09-27T16:55:56.8201193Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163759.xml (deflated 40%)
2022-09-27T16:55:56.8201638Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163818.xml (deflated 41%)
2022-09-27T16:55:56.8202066Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163829.xml (deflated 41%)
2022-09-27T16:55:56.8202488Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163839.xml (deflated 41%)
2022-09-27T16:55:56.8202918Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163850.xml (deflated 40%)
2022-09-27T16:55:56.8203341Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163900.xml (deflated 40%)
2022-09-27T16:55:56.8203770Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163911.xml (deflated 42%)
2022-09-27T16:55:56.8204200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163921.xml (deflated 41%)
2022-09-27T16:55:56.8204624Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163931.xml (deflated 40%)
2022-09-27T16:55:56.8205055Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163940.xml (deflated 40%)
2022-09-27T16:55:56.8205476Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163950.xml (deflated 40%)
2022-09-27T16:55:56.8205904Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927163959.xml (deflated 40%)
2022-09-27T16:55:56.8206333Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164009.xml (deflated 41%)
2022-09-27T16:55:56.8206749Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164019.xml (deflated 41%)
2022-09-27T16:55:56.8207177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164029.xml (deflated 40%)
2022-09-27T16:55:56.8207741Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164038.xml (deflated 40%)
2022-09-27T16:55:56.8208165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164049.xml (deflated 42%)
2022-09-27T16:55:56.8208580Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164053.xml (deflated 42%)
2022-09-27T16:55:56.8209003Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164057.xml (deflated 40%)
2022-09-27T16:55:56.8209430Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164108.xml (deflated 40%)
2022-09-27T16:55:56.8209858Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164115.xml (deflated 40%)
2022-09-27T16:55:56.8210282Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164122.xml (deflated 40%)
2022-09-27T16:55:56.8210710Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164132.xml (deflated 41%)
2022-09-27T16:55:56.8211139Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164141.xml (deflated 40%)
2022-09-27T16:55:56.8211565Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164148.xml (deflated 41%)
2022-09-27T16:55:56.8212031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164151.xml (deflated 41%)
2022-09-27T16:55:56.8212462Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164154.xml (deflated 40%)
2022-09-27T16:55:56.8212876Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164205.xml (deflated 41%)
2022-09-27T16:55:56.8213308Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164208.xml (deflated 41%)
2022-09-27T16:55:56.8213732Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164210.xml (deflated 40%)
2022-09-27T16:55:56.8214157Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164220.xml (deflated 42%)
2022-09-27T16:55:56.8214586Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164225.xml (deflated 41%)
2022-09-27T16:55:56.8215008Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164232.xml (deflated 40%)
2022-09-27T16:55:56.8215433Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164241.xml (deflated 41%)
2022-09-27T16:55:56.8215862Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164249.xml (deflated 40%)
2022-09-27T16:55:56.8216283Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164256.xml (deflated 40%)
2022-09-27T16:55:56.8216703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164303.xml (deflated 40%)
2022-09-27T16:55:56.8217119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164313.xml (deflated 42%)
2022-09-27T16:55:56.8217543Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164315.xml (deflated 42%)
2022-09-27T16:55:56.8217971Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164318.xml (deflated 42%)
2022-09-27T16:55:56.8218456Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164320.xml (deflated 42%)
2022-09-27T16:55:56.8218881Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164323.xml (deflated 40%)
2022-09-27T16:55:56.8219311Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164349.xml (deflated 41%)
2022-09-27T16:55:56.8219746Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164414.xml (deflated 42%)
2022-09-27T16:55:56.8220174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164416.xml (deflated 41%)
2022-09-27T16:55:56.8220605Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164419.xml (deflated 41%)
2022-09-27T16:55:56.8221039Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164421.xml (deflated 41%)
2022-09-27T16:55:56.8221468Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164424.xml (deflated 41%)
2022-09-27T16:55:56.8221884Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164426.xml (deflated 41%)
2022-09-27T16:55:56.8222360Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164435.xml (deflated 41%)
2022-09-27T16:55:56.8222803Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164444.xml (deflated 41%)
2022-09-27T16:55:56.8223236Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164454.xml (deflated 40%)
2022-09-27T16:55:56.8223669Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164502.xml (deflated 42%)
2022-09-27T16:55:56.8224098Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164507.xml (deflated 42%)
2022-09-27T16:55:56.8224526Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164509.xml (deflated 41%)
2022-09-27T16:55:56.8224954Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164512.xml (deflated 41%)
2022-09-27T16:55:56.8225389Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164514.xml (deflated 40%)
2022-09-27T16:55:56.8225819Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164521.xml (deflated 40%)
2022-09-27T16:55:56.8226237Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164529.xml (deflated 41%)
2022-09-27T16:55:56.8226668Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164531.xml (deflated 42%)
2022-09-27T16:55:56.8227101Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164534.xml (deflated 42%)
2022-09-27T16:55:56.8227530Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164538.xml (deflated 40%)
2022-09-27T16:55:56.8227961Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164548.xml (deflated 40%)
2022-09-27T16:55:56.8228392Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164558.xml (deflated 40%)
2022-09-27T16:55:56.8228821Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164608.xml (deflated 40%)
2022-09-27T16:55:56.8229317Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164617.xml (deflated 39%)
2022-09-27T16:55:56.8229750Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164627.xml (deflated 40%)
2022-09-27T16:55:56.8230180Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164634.xml (deflated 40%)
2022-09-27T16:55:56.8230615Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164641.xml (deflated 40%)
2022-09-27T16:55:56.8231030Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164656.xml (deflated 41%)
2022-09-27T16:55:56.8231460Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164659.xml (deflated 41%)
2022-09-27T16:55:56.8231887Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164701.xml (deflated 41%)
2022-09-27T16:55:56.8232312Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164704.xml (deflated 41%)
2022-09-27T16:55:56.8232740Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164706.xml (deflated 41%)
2022-09-27T16:55:56.8233208Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164708.xml (deflated 41%)
2022-09-27T16:55:56.8233645Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164711.xml (deflated 41%)
2022-09-27T16:55:56.8234071Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164713.xml (deflated 41%)
2022-09-27T16:55:56.8234506Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164715.xml (deflated 41%)
2022-09-27T16:55:56.8234931Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164718.xml (deflated 41%)
2022-09-27T16:55:56.8235346Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164720.xml (deflated 40%)
2022-09-27T16:55:56.8235778Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164729.xml (deflated 41%)
2022-09-27T16:55:56.8236206Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164731.xml (deflated 40%)
2022-09-27T16:55:56.8236632Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164741.xml (deflated 41%)
2022-09-27T16:55:56.8237061Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164743.xml (deflated 40%)
2022-09-27T16:55:56.8237484Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164752.xml (deflated 40%)
2022-09-27T16:55:56.8237909Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164801.xml (deflated 41%)
2022-09-27T16:55:56.8238338Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164804.xml (deflated 41%)
2022-09-27T16:55:56.8238763Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164806.xml (deflated 41%)
2022-09-27T16:55:56.8239189Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164808.xml (deflated 41%)
2022-09-27T16:55:56.8239657Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164811.xml (deflated 40%)
2022-09-27T16:55:56.8240084Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164821.xml (deflated 41%)
2022-09-27T16:55:56.8240665Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164832.xml (deflated 41%)
2022-09-27T16:55:56.8241094Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164834.xml (deflated 41%)
2022-09-27T16:55:56.8241525Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164836.xml (deflated 41%)
2022-09-27T16:55:56.8241952Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164839.xml (deflated 43%)
2022-09-27T16:55:56.8242385Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164841.xml (deflated 42%)
2022-09-27T16:55:56.8242813Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164844.xml (deflated 42%)
2022-09-27T16:55:56.8243240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164846.xml (deflated 42%)
2022-09-27T16:55:56.8243664Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164849.xml (deflated 42%)
2022-09-27T16:55:56.8244158Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164851.xml (deflated 40%)
2022-09-27T16:55:56.8244587Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164900.xml (deflated 40%)
2022-09-27T16:55:56.8245012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164908.xml (deflated 40%)
2022-09-27T16:55:56.8245445Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164918.xml (deflated 42%)
2022-09-27T16:55:56.8245874Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164920.xml (deflated 42%)
2022-09-27T16:55:56.8246302Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164923.xml (deflated 42%)
2022-09-27T16:55:56.8246736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164925.xml (deflated 42%)
2022-09-27T16:55:56.8247168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164927.xml (deflated 42%)
2022-09-27T16:55:56.8247596Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164930.xml (deflated 42%)
2022-09-27T16:55:56.8248026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164932.xml (deflated 41%)
2022-09-27T16:55:56.8248450Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164942.xml (deflated 41%)
2022-09-27T16:55:56.8248863Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164945.xml (deflated 40%)
2022-09-27T16:55:56.8249294Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927164954.xml (deflated 41%)
2022-09-27T16:55:56.8249723Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165004.xml (deflated 40%)
2022-09-27T16:55:56.8250149Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220927165018.xml (deflated 40%)
2022-09-27T16:55:56.8250547Z   adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 72%)
2022-09-27T16:55:56.8250862Z   adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 71%)
2022-09-27T16:55:56.8251182Z   adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 80%)
2022-09-27T16:55:56.8251530Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 81%)
2022-09-27T16:55:56.8251874Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 79%)
2022-09-27T16:55:56.8252245Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 67%)
2022-09-27T16:55:56.8252510Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-09-27T16:55:56.8273623Z ##[group]Run # Remove any previous test reports if they exist
2022-09-27T16:55:56.8273833Z # Remove any previous test reports if they exist
2022-09-27T16:55:56.8273959Z rm -f usage-log-*.zip
2022-09-27T16:55:56.8274172Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-09-27T16:55:56.8274319Z # so check to see if the file exists first
2022-09-27T16:55:56.8274449Z if [ -f 'usage_log.txt' ]; then
2022-09-27T16:55:56.8274608Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-09-27T16:55:56.8274701Z fi
2022-09-27T16:55:56.8286894Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:55:56.8287055Z env:
2022-09-27T16:55:56.8287181Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:56.8287296Z   GPU_FLAG: --gpus all
2022-09-27T16:55:56.8287524Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118
2022-09-27T16:55:56.8287611Z ##[endgroup]
2022-09-27T16:55:56.9376922Z   adding: usage_log.txt (deflated 95%)
2022-09-27T16:55:56.9425932Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:55:56.9426036Z with:
2022-09-27T16:55:56.9426200Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:55:56.9426318Z   retention-days: 14
2022-09-27T16:55:56.9426446Z   if-no-files-found: warn
2022-09-27T16:55:56.9426568Z   path: test-jsons-*.zip
2022-09-27T16:55:56.9426673Z   name: artifact
2022-09-27T16:55:56.9426774Z   s3-bucket: gha-artifacts
2022-09-27T16:55:56.9426884Z   region: us-east-1
2022-09-27T16:55:56.9426976Z env:
2022-09-27T16:55:56.9427099Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:56.9427214Z   GPU_FLAG: --gpus all
2022-09-27T16:55:56.9427318Z ##[endgroup]
2022-09-27T16:55:57.3465078Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:55:57.3465470Z With the provided path, there will be 1 file uploaded
2022-09-27T16:55:57.3465848Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:55:57.3475896Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118.zip
2022-09-27T16:55:57.5001139Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118.zip
2022-09-27T16:55:57.5161862Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:55:57.5162136Z with:
2022-09-27T16:55:57.5162417Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:55:57.5162718Z   retention-days: 14
2022-09-27T16:55:57.5162974Z   if-no-files-found: error
2022-09-27T16:55:57.5163255Z   path: test-reports-*.zip
2022-09-27T16:55:57.5163515Z   name: artifact
2022-09-27T16:55:57.5163752Z   s3-bucket: gha-artifacts
2022-09-27T16:55:57.5164018Z   region: us-east-1
2022-09-27T16:55:57.5164257Z env:
2022-09-27T16:55:57.5164483Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:57.5164748Z   GPU_FLAG: --gpus all
2022-09-27T16:55:57.5164995Z ##[endgroup]
2022-09-27T16:55:57.9159557Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:55:57.9159995Z With the provided path, there will be 1 file uploaded
2022-09-27T16:55:57.9160931Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:55:57.9171812Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118.zip
2022-09-27T16:55:58.0720494Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118.zip
2022-09-27T16:55:58.0870413Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-09-27T16:55:58.0870707Z with:
2022-09-27T16:55:58.0870990Z   s3-prefix: pytorch/pytorch/3133193930/2/artifact

2022-09-27T16:55:58.0871275Z   retention-days: 14
2022-09-27T16:55:58.0871547Z   if-no-files-found: ignore
2022-09-27T16:55:58.0871822Z   path: usage-log-*.zip
2022-09-27T16:55:58.0872064Z   name: artifact
2022-09-27T16:55:58.0872318Z   s3-bucket: gha-artifacts
2022-09-27T16:55:58.0872581Z   region: us-east-1
2022-09-27T16:55:58.0872807Z env:
2022-09-27T16:55:58.0873028Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:58.0873292Z   GPU_FLAG: --gpus all
2022-09-27T16:55:58.0873539Z ##[endgroup]
2022-09-27T16:55:58.4856360Z NOTE: s3-prefix specified, ignoring name parameter
2022-09-27T16:55:58.4857208Z With the provided path, there will be 1 file uploaded
2022-09-27T16:55:58.4857779Z Uploading to s3 prefix: pytorch/pytorch/3133193930/2/artifact
2022-09-27T16:55:58.4867480Z Starting upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118.zip
2022-09-27T16:55:58.6560172Z Finished upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_8576432118.zip
2022-09-27T16:55:58.6717512Z ##[group]Run set -x
2022-09-27T16:55:58.6717790Z set -x
2022-09-27T16:55:58.6718090Z python3 -m pip install -r requirements.txt
2022-09-27T16:55:58.6718429Z python3 -m pip install boto3==1.19.12
2022-09-27T16:55:58.6718825Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-09-27T16:55:58.6732576Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:55:58.6732875Z env:
2022-09-27T16:55:58.6733113Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:55:58.6733373Z   GPU_FLAG: --gpus all
2022-09-27T16:55:58.6733643Z   AWS_DEFAULT_REGION: us-east-1
2022-09-27T16:55:58.6733907Z   BRANCH: pull/85462
2022-09-27T16:55:58.6734150Z   TEST_CONFIG: distributed
2022-09-27T16:55:58.6734405Z   SHARD_NUMBER: 1
2022-09-27T16:55:58.6734721Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T16:55:58.6735018Z   PR_NUMBER: 85462
2022-09-27T16:55:58.6735277Z   PYTORCH_RETRY_TEST_CASES: 1
2022-09-27T16:55:58.6735599Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-09-27T16:55:58.6735894Z   SHA1: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T16:55:58.6736159Z   TAG: 
2022-09-27T16:55:58.6736388Z   WORKFLOW_ID: 3133193930
2022-09-27T16:55:58.6736800Z   GITHUB_TOKEN: ***
2022-09-27T16:55:58.6737060Z   GHA_WORKFLOW_JOB_ID: 8576432118
2022-09-27T16:55:58.6737313Z ##[endgroup]
2022-09-27T16:55:58.6766694Z + python3 -m pip install -r requirements.txt
2022-09-27T16:55:58.9652235Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:55:59.0463508Z Collecting astunparse
2022-09-27T16:55:59.0707703Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-09-27T16:55:59.1028902Z Collecting expecttest
2022-09-27T16:55:59.1071229Z   Downloading expecttest-0.1.3-py3-none-any.whl (6.5 kB)
2022-09-27T16:55:59.1473404Z Collecting future
2022-09-27T16:55:59.1516707Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-09-27T16:56:01.1105909Z Collecting hypothesis
2022-09-27T16:56:01.1171905Z   Downloading hypothesis-6.54.6-py3-none-any.whl (390 kB)
2022-09-27T16:56:01.9389671Z Collecting numpy
2022-09-27T16:56:01.9447641Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-09-27T16:56:02.2925648Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-09-27T16:56:02.4167531Z Collecting pyyaml
2022-09-27T16:56:02.4221835Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-09-27T16:56:02.4434825Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-09-27T16:56:02.4617401Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-09-27T16:56:02.5243928Z Collecting six
2022-09-27T16:56:02.5296574Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-09-27T16:56:02.5647876Z Collecting types-dataclasses
2022-09-27T16:56:02.5690443Z   Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB)
2022-09-27T16:56:02.6120960Z Collecting typing_extensions
2022-09-27T16:56:02.6170722Z   Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB)
2022-09-27T16:56:02.6765151Z Collecting sympy
2022-09-27T16:56:02.6847300Z   Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB)
2022-09-27T16:56:02.9151759Z Collecting wheel<1.0,>=0.23.0
2022-09-27T16:56:02.9234864Z   Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
2022-09-27T16:56:02.9777915Z Collecting attrs>=19.2.0
2022-09-27T16:56:02.9851358Z   Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
2022-09-27T16:56:03.0698784Z Collecting sortedcontainers<3.0.0,>=2.1.0
2022-09-27T16:56:03.0740760Z   Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
2022-09-27T16:56:03.1068897Z Collecting exceptiongroup>=1.0.0rc8; python_version < "3.11"
2022-09-27T16:56:03.1171047Z   Downloading exceptiongroup-1.0.0rc9-py3-none-any.whl (12 kB)
2022-09-27T16:56:03.1283592Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-09-27T16:56:03.1295116Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-09-27T16:56:03.1531743Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-09-27T16:56:03.1547561Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-09-27T16:56:03.1837861Z Collecting mpmath>=0.19
2022-09-27T16:56:03.1923465Z   Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
2022-09-27T16:56:03.2191808Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-09-27T16:56:03.3671807Z Installing collected packages: six, wheel, astunparse, expecttest, future, attrs, sortedcontainers, exceptiongroup, hypothesis, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy
2022-09-27T16:56:03.4083258Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:56:03.4084142Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:56:03.4395955Z     Running setup.py install for future: started
2022-09-27T16:56:04.1051587Z     Running setup.py install for future: finished with status 'done'
2022-09-27T16:56:04.4078640Z   WARNING: The script hypothesis is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:56:04.4079338Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:56:06.3750197Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:56:06.3750923Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:56:15.2801886Z   WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-09-27T16:56:15.2802568Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-09-27T16:56:15.3236357Z Successfully installed astunparse-1.6.3 attrs-22.1.0 exceptiongroup-1.0.0rc9 expecttest-0.1.3 future-0.18.2 hypothesis-6.54.6 mpmath-1.2.1 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.3.0 wheel-0.37.1
2022-09-27T16:56:15.3994283Z + python3 -m pip install boto3==1.19.12
2022-09-27T16:56:15.6900889Z Defaulting to user installation because normal site-packages is not writeable
2022-09-27T16:56:16.6245072Z Collecting boto3==1.19.12
2022-09-27T16:56:16.6438715Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-09-27T16:56:17.7552806Z Collecting botocore<1.23.0,>=1.22.12
2022-09-27T16:56:17.7690470Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-09-27T16:56:18.0066830Z Collecting jmespath<1.0.0,>=0.7.1
2022-09-27T16:56:18.0139192Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-09-27T16:56:18.0633372Z Collecting s3transfer<0.6.0,>=0.5.0
2022-09-27T16:56:18.0675945Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-09-27T16:56:18.0843246Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-09-27T16:56:18.1524605Z Collecting python-dateutil<3.0.0,>=2.1
2022-09-27T16:56:18.1576421Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-09-27T16:56:18.1765753Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-09-27T16:56:18.3602259Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
2022-09-27T16:56:19.2607005Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-09-27T16:56:19.3146621Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-09-27T16:56:29.0969670Z [scribe] Scribe access token not provided, sending report via boto3...
2022-09-27T16:56:29.0969943Z 
2022-09-27T16:56:29.0975740Z ----- Historic stats comparison result ------
2022-09-27T16:56:29.0976185Z 
2022-09-27T16:56:29.0976752Z     job: linux-bionic-cuda11.6-py3.10-gcc7
2022-09-27T16:56:29.0977150Z     commit: 52424e2bf38e454d535881fed9628d3e20f4f944
2022-09-27T16:56:29.0977353Z 
2022-09-27T16:56:29.0977583Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-09-27T16:56:29.0980422Z 
2022-09-27T16:56:29.0980875Z     : (master)
2022-09-27T16:56:29.0981307Z     |
2022-09-27T16:56:29.0981826Z     | * 52424e2bf3 (HEAD)            total time  1236.70s
2022-09-27T16:56:29.0982391Z     | |
2022-09-27T16:56:29.0982868Z     | : (4 commits)
2022-09-27T16:56:29.0983176Z     |/
2022-09-27T16:56:29.0983744Z     * c7c2578f93 (base)   9 reports, total time  3338.43s ± 1882.56s
2022-09-27T16:56:29.0984195Z     * 99ad8a3048          9 reports, total time  3402.28s ± 1860.32s
2022-09-27T16:56:29.0984680Z     * 34296e2f4c          9 reports, total time  3340.35s ± 1886.81s
2022-09-27T16:56:29.0988727Z     * 4523ac7aa1          9 reports, total time  3366.30s ± 1853.68s
2022-09-27T16:56:29.0989210Z     * f21e77d9a6          9 reports, total time  3413.29s ± 1861.08s
2022-09-27T16:56:29.0989666Z     * 26a861cb27          9 reports, total time  3325.52s ± 1844.86s
2022-09-27T16:56:29.0990156Z     * 56a41b5998          9 reports, total time  3452.70s ± 1976.19s
2022-09-27T16:56:29.0990578Z     * 1910c5847e          9 reports, total time  3444.18s ± 2043.46s
2022-09-27T16:56:29.0991014Z     * caa0ab557d          9 reports, total time  3312.06s ± 1824.07s
2022-09-27T16:56:29.0991323Z     * 0336308be5          0 reports
2022-09-27T16:56:29.0991548Z     |
2022-09-27T16:56:29.0991759Z     :
2022-09-27T16:56:29.0991894Z 
2022-09-27T16:56:29.0992064Z Removed  (across  953 suites)     0 tests, totaling     0.00s
2022-09-27T16:56:29.0992417Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-09-27T16:56:29.0993060Z Added    (across   10 suites)   330 tests, totaling +6127.57s
2022-09-27T16:56:29.1532414Z Prepare all required actions
2022-09-27T16:56:29.1560551Z ##[group]Run ./.github/actions/teardown-linux
2022-09-27T16:56:29.1560858Z with:
2022-09-27T16:56:29.1561061Z env:
2022-09-27T16:56:29.1561310Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:56:29.1561589Z   GPU_FLAG: --gpus all
2022-09-27T16:56:29.1561829Z ##[endgroup]
2022-09-27T16:56:29.1581869Z ##[group]Run set -eou pipefail
2022-09-27T16:56:29.1582197Z set -eou pipefail
2022-09-27T16:56:29.1582467Z 
2022-09-27T16:56:29.1582787Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-09-27T16:56:29.1583150Z for _ in $(seq 1440); do
2022-09-27T16:56:29.1583476Z     # Break if no ssh session exists anymore
2022-09-27T16:56:29.1583779Z     if [ "$(who)" = "" ]; then
2022-09-27T16:56:29.1584046Z       break
2022-09-27T16:56:29.1584294Z     fi
2022-09-27T16:56:29.1584547Z     echo "."
2022-09-27T16:56:29.1584823Z     sleep 5
2022-09-27T16:56:29.1585071Z done
2022-09-27T16:56:29.1598577Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:56:29.1598892Z env:
2022-09-27T16:56:29.1599145Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:56:29.1599412Z   GPU_FLAG: --gpus all
2022-09-27T16:56:29.1599674Z ##[endgroup]
2022-09-27T16:56:29.1629816Z Holding runner for 2 hours until all ssh sessions have logged out
2022-09-27T16:56:29.1709622Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-09-27T16:56:29.1710053Z # ignore expansion of "docker ps -q" since it could be empty
2022-09-27T16:56:29.1710399Z # shellcheck disable=SC2046
2022-09-27T16:56:29.1710695Z docker stop $(docker ps -q) || true
2022-09-27T16:56:29.1711010Z # Prune all of the docker images
2022-09-27T16:56:29.1711305Z docker system prune -af
2022-09-27T16:56:29.1723947Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-27T16:56:29.1724261Z env:
2022-09-27T16:56:29.1724506Z   GIT_DEFAULT_BRANCH: master
2022-09-27T16:56:29.1724763Z   GPU_FLAG: --gpus all
2022-09-27T16:56:29.1725014Z ##[endgroup]
2022-09-27T16:56:30.1869462Z 28fd6574525b
2022-09-27T16:56:30.9503727Z Deleted Containers:
2022-09-27T16:56:30.9504126Z 28fd6574525b362559bd9225008ae2c88d5908bbdc045faa284102984d334c89
2022-09-27T16:56:30.9504355Z 
2022-09-27T16:56:36.0861407Z Deleted Images:
2022-09-27T16:56:36.0862462Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:e66cf5fa0a4d4ed512901b12ccdab95cca946a29
2022-09-27T16:56:36.0863863Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:9bb261bc4d8aeb82a71b1f0709da9c979e85a12a79c4a85c3fe3adddddcb2663
2022-09-27T16:56:36.0864545Z deleted: sha256:1565775a6d0c052a41180f67487ffe62db8903a6db8b459487e830a767b885e7
2022-09-27T16:56:36.0865021Z deleted: sha256:3e4cb2c2b5f9e2c80e23a8d896fa974adc4b0e3a54cc06c0a8afb922880fdac8
2022-09-27T16:56:36.0865471Z deleted: sha256:d0ad3421a88e79ceec8792dd7f305a7f9da57daa0119b35d0a37114fd2a8dcf9
2022-09-27T16:56:36.0865922Z deleted: sha256:701e765e83a6966eae0a6dd8fc686dd1787c602cf538ef7dea4100368068fcc9
2022-09-27T16:56:36.0866364Z deleted: sha256:8e85fd60215cbc7be6e9f6423e1f87f0e07c0c672606d79f499dbc625e3eda75
2022-09-27T16:56:36.0866809Z deleted: sha256:e5bde08e611f82f97b690c9fa678975e18d51c1d1bc1c8ddef0588f1a022d639
2022-09-27T16:56:36.0867226Z deleted: sha256:68800e2bce8407a9b0b64d467c217a2bb27e85fd2f3ab0c5793759d5443bf962
2022-09-27T16:56:36.0867654Z deleted: sha256:2457e2e8215ddc219f679a7f957fdc3c639147fc569a0cc1bc9f3a3a97ecd0b2
2022-09-27T16:56:36.0868095Z deleted: sha256:b9c61d9be1eb46057ce27c51ac051ba0ba53d440bda828612cfc06ae78352d7a
2022-09-27T16:56:36.0868544Z deleted: sha256:73dd107fbbff1e853191e9dcb9b75847329af3c80a5883c3ae039e14fa4caa0a
2022-09-27T16:56:36.0868985Z deleted: sha256:fe97c2c62ffb15e65ed2751c4c9069d62e91f204ff1c30980d18680617c5ae40
2022-09-27T16:56:36.0869759Z deleted: sha256:3393d072706e15983b6dba8491dcfaf03de10df9be7d9df9a80ef0dedf384b3e
2022-09-27T16:56:36.0870200Z deleted: sha256:328eaea416a7110b96dd1c4421ae76dac29d151937f13f647291840207613cfd
2022-09-27T16:56:36.0870617Z deleted: sha256:1f62c29a7ac809a8add06b0bc14387e38ec12e88034824daccfdd78a0ee24d00
2022-09-27T16:56:36.0871056Z deleted: sha256:a364f929327c0e6baf5da371c2e514c56a7dc2fcbd3f7f2229d49bf2fb3a2f27
2022-09-27T16:56:36.0871495Z deleted: sha256:14a20e2c555e8aea13a33ca9c7d20333ce850c23d3a6726d7b102a426c2bb100
2022-09-27T16:56:36.0871963Z deleted: sha256:865844d2dbf2eb79ca92ace32b76fd966b142ecccd91a6190a256f4d278fed74
2022-09-27T16:56:36.0872372Z deleted: sha256:6a56212e72bab94d436ef418303481e278e79a012bf88cc727231093221416e7
2022-09-27T16:56:36.0872793Z deleted: sha256:8ff11e3f91c81bfe67a7da7b03306cdc936f730f44e7dbc401299c81679f3a20
2022-09-27T16:56:36.0873217Z deleted: sha256:e3ebc0052e1300157e36aa327172ad1098717918585d1494c4efc4afdbb8ecbe
2022-09-27T16:56:36.0873638Z deleted: sha256:4b7db58c33bbdbe2e491f721c5a9724ef07d9a720d77bd517c1f5c8fad212ba2
2022-09-27T16:56:36.0874090Z deleted: sha256:1e68b2d3e3c7bf741f5f060b3a88095db0d5eefd841ce4e86b3c466f882997e6
2022-09-27T16:56:36.0874530Z deleted: sha256:4da2a66a2b27b034b75aa5a024cf7b8cfe1f8762b0725921f8b00cb3a0505759
2022-09-27T16:56:36.0874981Z deleted: sha256:37c6c713aea14d8a4a5f0dfd63f80f04633b0d6f8e7baed0a1feba47d709cbeb
2022-09-27T16:56:36.0875413Z deleted: sha256:c636e275b4e8c2d9022e72ee8d8528006ef92ad2fab903ba909244c2b9aa4bb0
2022-09-27T16:56:36.0875858Z deleted: sha256:2b912a242a1b69ebde3bf40dfab4b693cfa93f9bcb0d8be6c698b44b4284a70d
2022-09-27T16:56:36.0876316Z deleted: sha256:e4ff425d4caf55cc2f5939271ab53c4382b45c5454ccb0e4fa62cbb730aa8ba7
2022-09-27T16:56:36.0876751Z deleted: sha256:f80b712f0269de04dedd4bb68f0a2103eb0ac5bb70e5da74a7fe0544c8a678af
2022-09-27T16:56:36.0877191Z deleted: sha256:c048927fdfe44aec1063f37c0137ad63b7537c23dd3051ad691232ce363048fa
2022-09-27T16:56:36.0877627Z deleted: sha256:85d0da5e4c90646a2049cf17e470deedd05b19ab1535e4932940055fc36e1c91
2022-09-27T16:56:36.0878071Z deleted: sha256:a94964edf2ce4a0b440f5d000d2ea97c6145adcb1ad201ade6214f02cd47626c
2022-09-27T16:56:36.0878492Z deleted: sha256:fc49e4f76476298cd45a4ef31767534bd2b6663c90683ccbb4d911ea77a93d56
2022-09-27T16:56:36.0878925Z deleted: sha256:25a90c74f1ddefff551a5575bbaeaf1fd45c71d629ef061f41a227f2813ebf62
2022-09-27T16:56:36.0879362Z deleted: sha256:9c67a539cd718b76768a481e6313606489c7afbdd2ed5f0d35f94a0c8161ff59
2022-09-27T16:56:36.0879765Z deleted: sha256:78cfd7e90437d18d294dd32f35632e1e0f4e93f98d2f74562b7a2a483d89e847
2022-09-27T16:56:36.0880660Z deleted: sha256:feccb37688a9f8b307ab298febd901ccc8920a2ec8e1660a0eb8de5b6e41a3b1
2022-09-27T16:56:36.0881483Z deleted: sha256:eeded198b25e05f7ca35680574d2863e48240eb544a4945d562943ed7c519eaa
2022-09-27T16:56:36.0882008Z deleted: sha256:eba0a8e367727026cab4d61ce7412eac92df541fa66418907501f777a611d01a
2022-09-27T16:56:36.0882429Z deleted: sha256:ab3963854dae2a8321e764d73ecfb89e77618f391cbe2698ab09c8129a35eb29
2022-09-27T16:56:36.0882847Z deleted: sha256:db198971615e367404b8c928ff87c4bc7e81b060fa78a8431f2124d37e546a69
2022-09-27T16:56:36.0883264Z deleted: sha256:116af78f4b87e494b37b6a65d9e3abdd828764632b0adf869c40db8a8afb7745
2022-09-27T16:56:36.0883680Z deleted: sha256:21479387e93277b4c3c8df1cbd39208f3c6aa4128b6f0eebaafa7e3b82d8a23c
2022-09-27T16:56:36.0884114Z deleted: sha256:5731c54a72f20a93dd164dc484ecd3fd0bf0f0cb98eec92e2382f99c0c43e1a6
2022-09-27T16:56:36.0884552Z deleted: sha256:8b6f381bcc3be3c07a12a8e528c8926c369875dccb5d18323007a3beb34cdf52
2022-09-27T16:56:36.0884990Z deleted: sha256:07dcc550108cf9d433a453ce3e351ac30e67c1ccbc5f7bc71b408e2fb4bcc6a9
2022-09-27T16:56:36.0885401Z deleted: sha256:129bdb873e79117f4e90135f0c6a58f775fcf596f4eb514b803771cef2da8278
2022-09-27T16:56:36.0885848Z deleted: sha256:2d49e3a81bd436bfd20fb4a849cdc98da82cb74afef3de38dda7a946d3fc4153
2022-09-27T16:56:36.0886304Z deleted: sha256:0ba4e259108e5311ddf6b79ae3a35f8f16a4004ef8817e50427baa3cc90ac081
2022-09-27T16:56:36.0886917Z deleted: sha256:c164403226561914f16becdeca65c54d20dba8dad414b062efc34c05c47bf725
2022-09-27T16:56:36.0887350Z deleted: sha256:cbe4006b2e6286d50c1b292fb71b69d5299d65f055285519eafc41eac3ef8a3c
2022-09-27T16:56:36.0887793Z deleted: sha256:edcec18dceb25f1a03ec20de4676464613e69072875a83f5c45e45a31aafc5b9
2022-09-27T16:56:36.0888231Z deleted: sha256:13c4f317ac4bb48997302756b8d5f8b602e835607c9806a1a5b200e9a0657d8a
2022-09-27T16:56:36.0888626Z deleted: sha256:57f043e380f4586c76968d6e062b50bac55254a5be7e80bea3c027a5bb316469
2022-09-27T16:56:36.0889040Z deleted: sha256:3e549931e0240b9aac25dc79ed6a6259863879a5c9bd20755f77cac27c1ab8c8
2022-09-27T16:56:36.0889278Z 
2022-09-27T16:56:36.0991556Z Total reclaimed space: 19.33GB
2022-09-27T16:56:36.1054254Z Post job cleanup.
2022-09-27T16:56:36.1092078Z Post job cleanup.
2022-09-27T16:56:36.2436489Z [command]/usr/bin/git version
2022-09-27T16:56:36.2487437Z git version 2.37.1
2022-09-27T16:56:36.2551793Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b6723c1c-17d0-4a09-be4d-56eff411dda2' before making global git config changes
2022-09-27T16:56:36.2552366Z Adding repository directory to the temporary git global config as a safe directory
2022-09-27T16:56:36.2561301Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-09-27T16:56:36.2609085Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-09-27T16:56:36.2648983Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-09-27T16:56:36.2971358Z Entering 'android/libs/fbjni'
2022-09-27T16:56:36.3012972Z Entering 'third_party/FP16'
2022-09-27T16:56:36.3057106Z Entering 'third_party/FXdiv'
2022-09-27T16:56:36.3096250Z Entering 'third_party/NNPACK'
2022-09-27T16:56:36.3136887Z Entering 'third_party/QNNPACK'
2022-09-27T16:56:36.3180411Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T16:56:36.3222357Z Entering 'third_party/XNNPACK'
2022-09-27T16:56:36.3276224Z Entering 'third_party/benchmark'
2022-09-27T16:56:36.3317517Z Entering 'third_party/cpuinfo'
2022-09-27T16:56:36.3360624Z Entering 'third_party/cub'
2022-09-27T16:56:36.3402400Z Entering 'third_party/cudnn_frontend'
2022-09-27T16:56:36.3450160Z Entering 'third_party/cutlass'
2022-09-27T16:56:36.3501140Z Entering 'third_party/eigen'
2022-09-27T16:56:36.3544817Z Entering 'third_party/fbgemm'
2022-09-27T16:56:36.3587650Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T16:56:36.3629136Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T16:56:36.3672256Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T16:56:36.3713137Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T16:56:36.3754898Z Entering 'third_party/flatbuffers'
2022-09-27T16:56:36.3798120Z Entering 'third_party/fmt'
2022-09-27T16:56:36.3839740Z Entering 'third_party/foxi'
2022-09-27T16:56:36.3881643Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T16:56:36.3922834Z Entering 'third_party/gloo'
2022-09-27T16:56:36.3963990Z Entering 'third_party/googletest'
2022-09-27T16:56:36.4007125Z Entering 'third_party/ideep'
2022-09-27T16:56:36.4048783Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T16:56:36.4091898Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T16:56:36.4140158Z Entering 'third_party/ios-cmake'
2022-09-27T16:56:36.4181249Z Entering 'third_party/ittapi'
2022-09-27T16:56:36.4222345Z Entering 'third_party/kineto'
2022-09-27T16:56:36.4263536Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T16:56:36.4305617Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T16:56:36.4350320Z Entering 'third_party/nccl/nccl'
2022-09-27T16:56:36.4391628Z Entering 'third_party/neon2sse'
2022-09-27T16:56:36.4433199Z Entering 'third_party/nlohmann'
2022-09-27T16:56:36.4476235Z Entering 'third_party/onnx'
2022-09-27T16:56:36.4532656Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T16:56:36.4576056Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T16:56:36.4619684Z Entering 'third_party/onnx-tensorrt'
2022-09-27T16:56:36.4662849Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T16:56:36.4710003Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T16:56:36.4752132Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T16:56:36.4794536Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T16:56:36.4841234Z Entering 'third_party/pocketfft'
2022-09-27T16:56:36.4884090Z Entering 'third_party/protobuf'
2022-09-27T16:56:36.4931494Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T16:56:36.4974356Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T16:56:36.5017378Z Entering 'third_party/psimd'
2022-09-27T16:56:36.5058986Z Entering 'third_party/pthreadpool'
2022-09-27T16:56:36.5101875Z Entering 'third_party/pybind11'
2022-09-27T16:56:36.5143902Z Entering 'third_party/python-enum'
2022-09-27T16:56:36.5186672Z Entering 'third_party/python-peachpy'
2022-09-27T16:56:36.5228561Z Entering 'third_party/python-six'
2022-09-27T16:56:36.5271122Z Entering 'third_party/sleef'
2022-09-27T16:56:36.5313477Z Entering 'third_party/tbb'
2022-09-27T16:56:36.5359188Z Entering 'third_party/tensorpipe'
2022-09-27T16:56:36.5403179Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T16:56:36.5445969Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T16:56:36.5487289Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T16:56:36.5529858Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T16:56:36.5570453Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T16:56:36.5614464Z Entering 'third_party/zstd'
2022-09-27T16:56:36.5676013Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-09-27T16:56:36.5705357Z http.https://github.com/.extraheader
2022-09-27T16:56:36.5716233Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-09-27T16:56:36.5755188Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-09-27T16:56:36.6072307Z Entering 'android/libs/fbjni'
2022-09-27T16:56:36.6098779Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6131136Z Entering 'third_party/FP16'
2022-09-27T16:56:36.6155376Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6188061Z Entering 'third_party/FXdiv'
2022-09-27T16:56:36.6212669Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6245683Z Entering 'third_party/NNPACK'
2022-09-27T16:56:36.6269622Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6302523Z Entering 'third_party/QNNPACK'
2022-09-27T16:56:36.6326822Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6358500Z Entering 'third_party/VulkanMemoryAllocator'
2022-09-27T16:56:36.6383338Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6415625Z Entering 'third_party/XNNPACK'
2022-09-27T16:56:36.6440133Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6485698Z Entering 'third_party/benchmark'
2022-09-27T16:56:36.6510143Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6543144Z Entering 'third_party/cpuinfo'
2022-09-27T16:56:36.6568338Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6601316Z Entering 'third_party/cub'
2022-09-27T16:56:36.6626656Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6659986Z Entering 'third_party/cudnn_frontend'
2022-09-27T16:56:36.6684626Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6722866Z Entering 'third_party/cutlass'
2022-09-27T16:56:36.6746678Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6786108Z Entering 'third_party/eigen'
2022-09-27T16:56:36.6810836Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6845629Z Entering 'third_party/fbgemm'
2022-09-27T16:56:36.6870897Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6904914Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-09-27T16:56:36.6929074Z http.https://github.com/.extraheader
2022-09-27T16:56:36.6960825Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-09-27T16:56:36.6984682Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7017621Z Entering 'third_party/fbgemm/third_party/googletest'
2022-09-27T16:56:36.7043177Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7074629Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-09-27T16:56:36.7099500Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7132677Z Entering 'third_party/flatbuffers'
2022-09-27T16:56:36.7157323Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7192093Z Entering 'third_party/fmt'
2022-09-27T16:56:36.7217782Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7250177Z Entering 'third_party/foxi'
2022-09-27T16:56:36.7274889Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7306680Z Entering 'third_party/gemmlowp/gemmlowp'
2022-09-27T16:56:36.7333202Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7366563Z Entering 'third_party/gloo'
2022-09-27T16:56:36.7391047Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7424297Z Entering 'third_party/googletest'
2022-09-27T16:56:36.7448867Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7481444Z Entering 'third_party/ideep'
2022-09-27T16:56:36.7506101Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7538646Z Entering 'third_party/ideep/mkl-dnn'
2022-09-27T16:56:36.7562631Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7595786Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-09-27T16:56:36.7620381Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7659812Z Entering 'third_party/ios-cmake'
2022-09-27T16:56:36.7685556Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7717084Z Entering 'third_party/ittapi'
2022-09-27T16:56:36.7742257Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7774126Z Entering 'third_party/kineto'
2022-09-27T16:56:36.7798113Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7830176Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-09-27T16:56:36.7855011Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7887925Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-09-27T16:56:36.7911545Z http.https://github.com/.extraheader
2022-09-27T16:56:36.7945764Z Entering 'third_party/nccl/nccl'
2022-09-27T16:56:36.7970385Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8002772Z Entering 'third_party/neon2sse'
2022-09-27T16:56:36.8026748Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8058557Z Entering 'third_party/nlohmann'
2022-09-27T16:56:36.8082848Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8115900Z Entering 'third_party/onnx'
2022-09-27T16:56:36.8140309Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8185212Z Entering 'third_party/onnx/third_party/benchmark'
2022-09-27T16:56:36.8209918Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8242455Z Entering 'third_party/onnx/third_party/pybind11'
2022-09-27T16:56:36.8266471Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8301264Z Entering 'third_party/onnx-tensorrt'
2022-09-27T16:56:36.8326689Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8357539Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-09-27T16:56:36.8382445Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8419432Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-09-27T16:56:36.8444294Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8476230Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-09-27T16:56:36.8500480Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8532640Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-09-27T16:56:36.8556405Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8594321Z Entering 'third_party/pocketfft'
2022-09-27T16:56:36.8618908Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8650737Z Entering 'third_party/protobuf'
2022-09-27T16:56:36.8674705Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8710592Z Entering 'third_party/protobuf/third_party/benchmark'
2022-09-27T16:56:36.8735049Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8766952Z Entering 'third_party/protobuf/third_party/googletest'
2022-09-27T16:56:36.8791216Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8826567Z Entering 'third_party/psimd'
2022-09-27T16:56:36.8851079Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8882584Z Entering 'third_party/pthreadpool'
2022-09-27T16:56:36.8907080Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8939598Z Entering 'third_party/pybind11'
2022-09-27T16:56:36.8964950Z http.https://github.com/.extraheader
2022-09-27T16:56:36.8996960Z Entering 'third_party/python-enum'
2022-09-27T16:56:36.9021428Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9053013Z Entering 'third_party/python-peachpy'
2022-09-27T16:56:36.9076988Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9109295Z Entering 'third_party/python-six'
2022-09-27T16:56:36.9134171Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9166842Z Entering 'third_party/sleef'
2022-09-27T16:56:36.9191481Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9224566Z Entering 'third_party/tbb'
2022-09-27T16:56:36.9250027Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9284818Z Entering 'third_party/tensorpipe'
2022-09-27T16:56:36.9310041Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9342882Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-09-27T16:56:36.9368479Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9400955Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-09-27T16:56:36.9425222Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9457319Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-09-27T16:56:36.9481082Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9514162Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-09-27T16:56:36.9538880Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9570405Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-09-27T16:56:36.9594113Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9629739Z Entering 'third_party/zstd'
2022-09-27T16:56:36.9654896Z http.https://github.com/.extraheader
2022-09-27T16:56:36.9952458Z Cleaning up orphan processes